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Findings  (cont'd) 


Objective 

We  inspected  the  F-35  Lightning  II 
Program  (F-35  Program)  at  Lockheed 
Martin,  Fort  Worth,  Texas,  for  conformity 
to  the  contractually  required  Aerospace 
Standard  (AS)9100,  "Quality  Management 
Systems  -  Requirements  for  Aviation, 

Space  and  Defense  Organizations." 

We  also  evaluated  corrective  actions 
taken  in  response  to  nonconformities, 
findings,  and  recommendations 
identified  in  DoD  Inspector  General  (IG) 
Report  No.  DODIG-2013-140, 

"Quality  Assurance  Assessment  of 
the  F-35  Lightning  II  Program," 

September  30,  2013,  to  determine  whether 
the  actions  taken  were  appropriate. 

Findings 

The  F-35  Program  generally  conformed  to 
requirements  and  showed  improvement  in 
quality  management  system  performance 
since  our  previous  evaluation;  however, 
challenges  still  remain,  as  evidenced  by 
57  nonconformities  to  the  AS9100  standard 
and  4  opportunities  for  improvement. 


B.  ensure  that  all  system  level  requirements  and 
capabilities  were  realized  and  verified, 

C.  create  an  independent  quality  assurance  organization, 
establish  its  roles  and  responsibilities,  and  ensure  it 
was  adequately  staffed  to  perform  effective  oversight 
for  the  F-35  Program, 

D.  ensure  that  Lockheed  Martin  was  taking  necessary 
steps  to  reduce  the  assembly  defect  rate  in  order  to 
meet  the  full  rate  production  goals, 

F.  ensure  that  Lockheed  Martin's  software  quality 
management  processes  were  performed  sufficiently 
to  prevent  software  defects,  and 

G.  ensure  that  Lockheed  Martin  flows  down  all 
contractual  requirements  to  its  subcontractors, 
evaluates  deliverables  for  contract  compliance,  and 
allows  minor  nonconformances  to  be  approved  only 
by  the  proper  authority. 

In  addition,  the  Defense  Contract  Management  Agency  (DCMA) 
did  not: 

E.  escalate  unresolved  Corrective  Action  Requests  (CARs) 
to  the  next  higher  level  as  appropriate  and  required 
by  its  policy,  for  effective  resolution. 

Recommendations 


The  Joint  Program  Office  (JPO)  did  not:  We  recommend  that  the  Joint  Program  Office: 


A.  ensure  the  program  made  sufficient 
progress  toward  full  compliance 
with  Public  Law  108-136,  "National 
Defense  Authorization  Act  for 
fiscal  year  2004,"  Section  802, 
"Quality  control  in  procurement  of 
aviation  Critical  Safety  Items  (CSIs) 
and  related  services,"  and  the 
Joint  Service  CSI  Instruction 
(SECNAVINST  4140.2),  "Management 
of  Aviation  Critical  Safety  Items," 


A.l.  ensure  that  the  F-35  CSI  Program  is  compliant  with 
Public  Law  108-136,  Section  802,  "Quality  control  in 
procurement  of  aviation  CSIs  and  related  services," 
and  the  Joint  Service  CSI  Instruction,  "Management 
of  Aviation  Critical  Safety  Items"; 

A. 2.  conduct  periodic  CSI  Program  evaluations  of 
Lockheed  Martin  and  its  suppliers  to  ensure 
compliance  with  public  law  and  the  Joint  Service 
CSI  Instruction; 


Visit  us  on  the  web  ot  www.dodig.mil 
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Recommendations  (cont'd) 

B.l.  clearly  define  contractual  criteria  for  the 

acceptance  of  all  future  and  fielded  aircraft  to 
ensure  that  aircraft  capabilities  are  verified; 

B. 2.  ensure  that  all  21  system-level  requirements 

that  may  not  be  met,  in  addition  to  the  risks 
associated  with  the  failure  to  meet  these 
requirements,  are  documented,  tracked, 
and  mitigated  using  the  established  risk 
management  process; 

C.  realign  the  quality  assurance  organization 
to  report  directly  to  the  Program 
Executive  Officer,  define  the  organization's 
roles  and  responsibilities,  and  staff  the 
organization  appropriately; 

D. l.  ensure  that  Lockheed  Martin  implements 

quality  improvement  initiatives  to  reduce 
the  assembly  defect  rate  to  meet  full  rate 
production  goals; 

D.2.  coordinate  with  DCMA  to  implement  an 

effective  root  cause  analysis  and  corrective 
action  process  in  order  to  reduce  assembly 
defect  rate; 

F.  work  with  Lockheed  Martin  to  ensure  software 
quality  management  systems  are  improved; 
metrics  should  be  reported  on  a  periodic  basis 
(for  example,  monthly)  to  evaluate  process 
improvement;  and 

G. l.  ensure  that  all  minor  nonconformances  are 

evaluated  and  approved  only  by  DCMA. 


We  recommend  that  the  DCMA: 

E.l.  review  all  unresolved  CARs  and  escalate  those 
that  meet  the  criteria  established  in  DCMA 
policies  and  instructions, 

E.2.  assess  all  CARs  that  were  not  properly  elevated 
and  assess  any  impact  on  the  product,  and 

G.2.  ensure  that  Lockheed  Martin  flows  down 

the  appropriate  technical  requirements  to  its 
subcontractors  and  receives  and  evaluates 
contract  deliverables  within  the  required 
time  frames. 

Management  Comments  and 
Our  Response 

On  January  20,  2015,  JPO  and  DCMA  provided 
comments  on  our  findings  and  recommendations. 

The  Joint  Program  Office  agreed  with 
six  recommendations  and  partially  agreed 
with  three  recommendations. 

JPO  partially  agreed  with  the  recommendation  to  track 
21  system-level  requirements,  which  it  acknowledged 
will  not  be  met.  However,  JPO  does  not  consider  the 
21  system-level  requirements  as  risks  and  did  not 
agree  to  track  them  in  its  formal  risk  management 
process.  We  disagree  with  this  approach  because 
a  final  determination  of  performance  has  not  been 
made  and  failure  to  track  the  risks  in  the  formal  risk 
management  process  prevents  an  identification  of  the 
plans  necessary  for  closure. 


ii  |  DODIG-2015-092  (Project  No.  D2014-DT0TAD-0004) 


Results  in  Brief 

F-35  Lightning  II  Program  Quality  Assurance  and 
Corrective  Action  Evaluation 


Management  Comments  and  Our  Response  (cont'd) 

JPO  partially  agreed  with  the  recommendation  to 
work  with  DCMA  to  implement  an  effective  root  cause 
analysis  and  corrective  action  process  to  reduce 
assembly  defects  to  meet  full-rate  production  goals. 
However,  JPO  stated  that  no  additional  changes  to 
corrective  action  processes  were  necessary.  We 
disagree  with  JPO's  response  because  additional  quality 
initiatives  are  required  to  meet  full-rate  production 
goals  and  DCMA's  involvement  is  necessary  to  ensure 
objectives  are  met.  Our  recommendation  was  for  JPO 
to  take  actions  to  identify  and  correct  the  cause  of  the 
program's  inability  to  reduce  defect  rates  to  support 
full-rate  production. 


JPO  also  partially  agreed  with  our  recommendation 
to  ensure  all  minor  nonconformances  are  evaluated 
and  approved  only  by  DCMA.  The  actions  that  JPO  is 
planning  to  meet  the  intent  of  the  recommendation. 

DCMA  agreed  with  all  three  of  our  recommendations. 
The  actions  that  DCMA  is  planning  to  take  meet  the 
intent  of  the  recommendations. 

Please  see  the  Recommendations  Table  on  the 
following  page. 
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Recommendations  Table 


Management 

Recommendations 
Requiring  Comment 

No  Additional 
Comments  Required 

Joint  Program  Office 

A.l,  B.l,  B.2,  C,  D.l,  D.2,  and  F 

A. 2  and  G.l 

Defense  Contract  Management  Agency 

E.l,  E.2,  and  G.2 

Please  provide  comments  by  April  13,  2015. 
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4800  MARK  CENTER  DRIVE 

ALEXANDRIA,  VIRGINIA  22350-1500 


March  11,  2015 

MEMORANDUM  FOR  PROGRAM  EXECUTIVE  OFFICE  JOINT  STRIKE  FIGHTER 

DIRECTOR,  DEFENSE  CONTRACT  MANAGEMENT  AGENCY 

SUBJECT:  F-35  Lightning  II  Program  Quality  Assurance  and  Corrective  Action 
Evaluation  (Report  No.  DODIG-2015-092) 

We  are  providing  this  report  for  review  and  comment.  The  DoD  Office  of  Inspector 
General  (OIG)  conducted  an  evaluation  of  the  F-35  Lightning  II  Program  (F-35  Program) 
for  conformity  to  the  contractually  required  Aerospace  Standard  (ASJ9100,  "Quality 
Management  Systems  -  Requirements  for  Aviation,  Space  and  Defense  Organizations,” 
and  to  determine  whether  the  program  took  appropriate  actions  on  findings  and 
recommendations  identified  in  the  DoD  Inspector  General  Report  No.  DODIG-2013-140, 
"Quality  Assurance  Assessment  of  the  F-35  Lightning  II  Program,"  September  30,  2013. 

We  conducted  this  evaluation  in  accordance  with  the  Council  of  Inspectors  General  on 
Integrity  and  Efficiency,  "Quality  Standards  for  Inspection  and  Evaluation." 

Our  evaluation  determined  that  the  F-35  Program  generally  conformed  to  requirements 
and  showed  improvement  in  its  quality  management  system  performance  with 
respect  to  our  previous  evaluation;  however,  challenges  still  remain,  as  evidenced  by 
57  nonconformities  to  the  AS9100  standard  and  4  opportunities  for  improvement. 

We  considered  management  comments  on  the  draft  of  this  report  when  preparing  the 
final  report.  DoD  Instruction  7650.03  requires  that  recommendations  be  resolved 
promptly.  Comments  from  the  Defense  Contract  Management  Agency  were  responsive, 
and  do  not  require  additional  comments.  Comments  from  the  F-35  Joint  Program  Office 
were  not  fully  responsive;  therefore,  we  request  further  comments  on  Recommendations 
A.l,  B.l,  B.2,  C,  D.l,  D.2,  and  F  by  April  13,  2015. 

Please  provide  comments  that  conform  to  the  requirements  for  DoD  Instruction  7650.03. 
If  possible,  please  send  a  PDF  file  containing  your  comments 

Copies  of  your  comments  must  have  the  actual  signature  of  the  authorizing  official 
for  your  organization.  We  cannot  accept  the  /Signed/  symbol  in  place  of  the  actual 
signature.  If  you  arrange  to  send  classified  comments  electronically,  you  must  send 
them  over  the  SECRET  Internet  Protocol  Router  Network  (SIPRNET). 


DODIG-2015-092  |  v 


We  appreciate  the  courtesies  extended  to  the  staff.  Please  direct  questions  to 


Jeputy  Inspector  General 
Policy  and  Oversight 


cc:  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 
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Introduction 


Objectives 

Our  objective  was  to  evaluate  and  verify  the  implementation  of  the  corrective 
actions  taken  by  the  F-35  Joint  Program  Office  (JPO),  Defense  Contract  Management 
Agency  (DCMA),  and  the  prime  contractor — Lockheed  Martin  Aeronautics 
Company  (Lockheed  Martin),  Fort  Worth,  Texas,  in  response  to  the  findings 
identified  in  DoD  Inspector  General  (IG)  Report  No.  DODIG-2013-140,  "Quality 
Assurance  Assessment  of  the  F-35  Lightning  II  Program,"  September  30,  2013. 

As  part  of  this  project,  we  also  reevaluated  Lockheed  Martin’s  implementation  of 
the  contractually  required  Aerospace  Standard  (AS)9100,  "Quality  Management 
Systems  -  Requirements  for  Aviation,  Space  and  Defense  Organizations," 
contractual  quality  assurance  clauses,  internal  quality  assurance  processes  and 
procedures,  and  public  law.  In  addition,  we  evaluated  JPO-  and  DCMA-assigned 
functions  and  responsibilities  related  to  oversight  and  monitoring  of 
F-35  contractors. 

Background 

The  F-35  Lightning  II  Program  (F-35  Program)  is  a  joint,  multiservice, 
multinational  acquisition  to  develop  and  field  a  next-generation  strike  fighter 
aircraft  for  the  Navy,  Air  Force,  Marine  Corps,  and  numerous  international 
partners  and  foreign  military  customers.  The  F-35  has  three  variants;  model 
F-35A — Conventional  Takeoff  and  Landing  (CTOL),  F-35B — Short  Takeoff  and 
Vertical  Landing  (STOVL),  and  F-35C — Carrier-Suitable  Variant  (CV).  In  addition 
to  hundreds  of  other  suppliers  worldwide,  Lockheed  Martin  has  two  principal 
subcontractors/suppliers,  Northrop  Grumman  Aerospace  Systems  (Northrop 
Grumman)  and  BAE  Systems  (BAE). 

The  F-35  Program  entered  the  System  Development  and  Demonstration  (SDD) 
phase  of  the  acquisition  lifecycle  in  October  2001;  however,  after  exceeding 
critical  cost  growth  thresholds  established  by  statute — a  condition  known  as  a 
critical1  Nunn-McCurdy  breach — the  original  Milestone  B  decision  was  rescinded  in 
June  2010  and  was  not  re-approved  until  March  2012,  after  program  restructuring. 
As  identified  in  the  F-35  acquisition  strategy,  the  F-35  Program  uses  development, 
production,  and  sustainment  activities  concurrently  with  11  separate  low-rate 
initial  production  (LRIP)  lot  deliveries.  Each  LRIP  lot  represents  an  increasing  level 
of  maturity  and  additional  system  capabilities  are  incrementally  delivered  as  the 
program  approaches  full-rate  production. 

1  Section  2433,  title  10,  United  States  Code  defines  a  critical  "Nunn-McCurdy"  unit  cost  breach  as  an  increase  of  at  least 
25  percent  over  the  program  acquisition  unit  cost  (PAUC)  or  average  procurement  unit  cost  for  the  program  (APUC)  or 
subprogram  as  shown  in  the  current  baseline  estimate  or  an  increase  of  at  least  50  percent  over  the  PAUC  or  APUC  for  the 
program  or  subprogram  as  shown  in  the  original  baseline  estimate. 
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As  of  September  2014,  Lockheed  Martin  delivered  a  total  of  106  aircraft 
including  the  majority  of  LRIP  lot  5  aircraft  and  was  planning  to  begin  delivery  of 
LRIP  lot  6  aircraft.  Lockheed  Martin  plans  to  deliver  LRIP  lots  6,  7,  and  8  aircraft 
with  Block  3I2  software  capabilities.  LRIP  lots  9  and  beyond  will  be  delivered  with 
additional  software  block  updates  to  enable  full  warfighting  capabilities. 

Summary  of  Previous  Evaluation  Results 

From  mid-2012  until  early  2013,  the  DoD  Office  of  Inspector  General  (OIG) 
performed  a  quality  management  system  evaluation  of  the  F-35  Program 
at  Lockheed  Martin  and  its  major  subcontractors.3  This  evaluation 
resulted  in  363  nonconformities4  to  the  AS9100  standard.  Eighty-seven  of 
these  nonconformities  were  written  against  Lockheed  Martin;  63  were 
documented  at  Lockheed  Martin,  Fort  Worth,  Texas  and  24  were  documented 
at  its  subcontractor  sites.  The  remaining  276  nonconformities  documented 
during  our  previous  evaluation  were  written  against  Northrop  Grumman; 

BAE;  Honeywell  Aerospace;  L-3  Display  Systems;  and  United  Technologies 
Corporation,  Aerospace  Systems  (UTAS),  and  were  not  included  in  this 
evaluation.  All  nonconformities  were  categorized  as  major,  minor,  or  an 
opportunity  for  improvement  (OFI),  in  accordance  with  AS9101,  "Quality 
Management  Systems  -  Audit  Requirements  for  Aviation,  Space,  and  Defense 
Organizations."  Of  the  63  nonconformities  that  were  directed  to  Lockheed  Martin 
at  its  Fort  Worth,  Texas,  facility,  23  were  categorized  as  major  and  40  were 
categorized  as  minor.  These  nonconformities  were  further  categorized  by  the 
applicable  top-level  AS9100  clauses  as  shown  in  Figure  1. 


2  Initial  training  and  warfighting  capabilities  were  included  in  software  blocks  prior  to  Block  31.  Block  31  includes  third 
generation  helmet  and  night  vision  capabilities.  Future  software  block  upgrades  will  include  full  warfighting  capabilities. 

3  DoD  IG  Report  No.  DODIG-2013-140,  "Quality  Assurance  Assessment  of  the  F-35  Lightning  II  Program." 

4  AS9101D  defines  a  major  nonconformity  as  "a  non-fulfillment  of  a  requirement  which  is  likely  to  result  in  the  failure  of 
the  quality  management  system  or  reduce  its  ability  to  assure  controlled  processes  or  compliant  products/services"  and 
a  minor  nonconformity  as  "a  non-fulfillment  of  a  requirement  which  is  not  likely  to  result  in  the  failure  of  the  quality 
management  system  or  reduce  its  ability  to  assure  controlled  processes  or  compliant  products/services." 
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Figure  1.  Nonconformities  Written  Against  Lockheed  Martin  at  its  Fort  Worth,  Texas, 
Facility  Documented  in  Report  No.  DODIG-2013-140 


Evaluation  Criteria 

We  performed  our  evaluation  based  on  the  AS9100C  standard,  which  the  contractor 
is  required  to  comply  with  in  accordance  with  the  F-35  contracts.  Additionally,  we 
evaluated  the  F-35  Program’s  compliance  with  applicable  statutory  and  regulatory 
requirements  to  include  the  Federal  Acquisition  Regulation;  Defense  Federal 
Acquisition  Regulation  Supplement;  DoD  Instruction  Interim  5000.02,  "Operation 
of  the  Defense  Acquisition  System,"  enclosure  1;  and  DoD  Manual  4140.01,  "DoD 
Supply  Chain  Materiel  Management  Procedures:  Materiel  Sourcing,”  volume  3. 
Furthermore,  we  evaluated  the  F-35  Program’s  implementation  of  aviation 
critical  safety  item  (CSI)  requirements.  Defense  Federal  Acquisition  Regulation 
Supplement  (DFARS)  Subpart  209.270,  "Aviation  and  Ship  Critical  Safety  Items” 
defines  an  aviation  CSI  as  "a  part,  an  assembly,  installation  equipment,  launch 
equipment,  recovery  equipment,  or  support  equipment  for  an  aircraft  or  aviation 
weapons  system"  that  contains  a  characteristic  such  that  any  failure,  malfunction, 
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or  absence  of  which  could  cause  a  catastrophic  or  critical  failure  resulting  in  the 
loss  of  life,  permanent  disability  or  major  injury,  loss  of  a  system,  or  significant 
equipment  damage.  As  part  of  this  evaluation,  we  performed  a  quality  assurance 
inspection  for  two  weeks  at  Lockheed  Martin,  Fort  Worth,  Texas  from  late 
September  2014  through  early  October  2014.  See  Appendix  A  for  more  details 
on  the  criteria  of  this  evaluation. 

Quality  Assurance  Evaluation  Process 

To  evaluate  JPO’s  management  of  the  F-35  quality  assurance  program  and 
Lockheed  Martin’s  implementation  of  corrective  actions  from  the  previous 
DoD  IG  report  (Report  No.  DODIG-2013-140),  we  performed  a  quality  assurance 
inspection  for  two  weeks  at  Lockheed  Martin,  Fort  Worth,  Texas  from  late 
September  2014  through  early  October  2014.  The  results  enabled  us  to  evaluate 
the  F-35  Program’s  conformity  to  the  contractually  required  AS9100C  quality 
management  system  standard,  contractual  quality  assurance  clauses,  and  internal 
Lockheed  Martin  quality  assurance  processes  and  procedures.  It  also  allowed 
us  to  evaluate  Lockheed  Martin’s  progress  on  its  quality  management  system 
performance.  Additionally,  we  evaluated  the  corrective  actions  taken  in  response 
to  the  nonconformities  discussed  in  the  previous  DoD  IG  evaluation  report  and 
verified  whether  the  nonconformities  were  adequately  resolved  or  on  schedule 
for  resolution. 

We  established  teams  of  engineering  and  subject  matter  experts  who  evaluated  the 
program  based  on  the  AS9100C  quality  management  system  standard.  The  subject 
matter  expert  teams  consisted  of  quality  assurance  engineers,  trained  and  certified 
in  AS9100,  who  had  an  average  of  17  years  of  quality  assurance  audit  experience. 
Additionally,  we  included  a  team  that  evaluated  the  Lockheed  Martin’s  aviation 
CSI  Program. 

Classification  and  Categorization  of  Nonconformities 

As  defined  by  the  AS9101  standard,  a  major  nonconformity  is  a  nonfulfillment  of  a 
requirement  that  is  likely  to  result  in  the  failure  of  the  quality  management  system 
or  reduce  its  ability  to  ensure  controlled  processes  or  compliant  products/services. 
A  minor  nonconformity  is  a  nonfulfillment  of  a  requirement  that  is  not  likely 
to  result  in  the  failure  of  the  quality  management  system  or  reduce  its  ability 
to  ensure  controlled  processes  or  compliant  products  or  services.  An  OFI  is  an 
industry  best  practice  where  a  specific  requirement  does  not  exist.  This  report 
focuses  on  the  major  nonconformities  documented  during  this  evaluation. 
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Organization  of  the  Report 

The  remainder  of  this  report  is  organized  in  three  sections,  the  first  of  which 
discusses  the  results  of  our  AS9100  inspection  of  Lockheed  Martin  conducted 
from  September  22  to  October  3,  2014,  the  second  documents  the  results  of 
our  evaluation  of  the  corrective  actions  taken  in  response  to  our  previous 
report  (DODIG-2013-140),  and  the  third  section  provides  a  summary  of  the  overall 
performance  of  the  F-35  program  quality  management  processes  by  comparing  the 
results  of  this  evaluation  to  the  results  of  our  previous  evaluation.  We  conclude 
the  report  with  a  discussion  of  our  overall  findings  and  recommendations. 


DODIG-2015-092  |  5 


Current  F-35  Program  Quality  Management  Performance 


Current  F-35  Program  Quality 
Management  Performance 

Results  of  AS9100  Conformance  Inspection  at 
Lockheed  Martin,  Fort  Worth,  Texas 

Our  inspection  at  Lockheed  Martin,  Fort  Worth,  Texas  from  late  September  2014 
through  early  October  2014  resulted  in  61  nonconformities  that  identified 
weaknesses  in  the  F-35  Program's  implementation  of  the  AS9100  quality 
management  system.  Of  these,  52  were  written  against  Lockheed  Martin  and  were 
categorized  as  24  major,  26  minor,  and  2  OFIs.  Four  nonconformities  were  written 
against  JPO:  two  major  and  two  OFIs.  In  addition,  five  were  written  against  DCMA: 
one  major  and  four  minor. 


Figure  2.  Lockheed  Martin  Nonconformities  Found  During  the  Evaluation 
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Lack  of  Commitment  to  Quality  Management  Objectives 

Management  Commitment  (5.1) 

In  accordance  with  the  F-35  Program  Quality  Management  Plan,  a  90-percent 
reduction  of  the  assembly  defect  rate  is  necessary  by  full-rate  production  in 
order  to  meet  the  goal  of  producing  one  aircraft  per  day.5  At  the  time  of  our 
inspection,  LRIP  lot  5  should  have  demonstrated  a  60-percent  reduction  in  defect 
rate;  however,  the  required  progress  toward  this  goal  could  not  be  determined 
because — in  accordance  with  conflicting  information  contained  in  the  quality 
management  plan — Lockheed  Martin  was  tracking  the  number  of  nonconforming 
material  records  rather  than  the  number  of  assembly  defects.  Nonconformance 
data  for  LRIP  lots  1  through  5  demonstrated  only  a  23-percent  reduction  of 
nonconforming  material  records.  The  lack  of  ability  to  accurately  determine  and 
reduce  assembly  defects  in  accordance  with  defined  program  objectives  may 
negatively  impact  the  full-rate  production  delivery  schedule. 

Insufficient  Software  Quality  Management  Rigor 

Competence,  Training,  and  Awareness  (6.2.2) 

We  identified  a  systemic  problem  with  Lockheed  Martin’s  software  engineer 
training  program.  Its  Learning  Management  System  and  Training  Requirements 
Matrix  only  captured  training  completion  status  and  did  not  include 
F-35  project-specific  training  plans  or  training  requirements  for  software 
developers.  The  development  of  software  intensive  systems,  such  as  the 
F-35  Program,  requires  software  development  integrated  product  team  members 
involved  in  the  design,  implementation,  and  testing  of  critical  software  to 
receive  specific  training  applicable  to  their  assigned  tasks.  Without  a  software 
development  training  program,  software  defects  could  increase  and  possibly 
degrade  product  performance,  reliability,  and  maintainability. 

Design  and  Development  Verification  (7.3.5) 

Lockheed  Martin  was  not  conducting  functional  configuration  audits  (FCAs)  and 
physical  configuration  audits  (PCAs)  of  the  Mission  System  and  Vehicle  System 
computer  software  configuration  items  (CSCIs).  The  intent  of  an  FCA  is  to  verify 
that  a  CSCI  (or  a  hardware  configuration  item  [HWCI])  meets  the  requirements 
documented  in  the  associated  specifications  by  examination  of  the  test  plans, 
procedures,  and  data.  A  PCA  is  formal  evaluation  of  the  as-built  configuration 
of  a  HWCI  or  a  CSCI  against  its  technical  documentation  to  establish  the 


5  The  F-35  Program  Quality  Management  Plan  defines  the  assembly  defect  rate  reduction  goal  with  respect  to  the 
SDD  baseline  and  prescribes  a  10-percent  reduction  per  LRIP  lot  in  order  to  achieve  a  90-percent  reduction  by  full 
rate  production. 
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product  baseline  (as-built  versus  as-designed).  Lockheed  Martin  required  all 
of  its  software  suppliers  to  verify  their  CSCIs  by  performing  FCAs  and  PCAs; 
however,  Lockheed  Martin  did  not  perform  FCAs/PCAs  for  internally  developed 
Mission  System  and  Vehicle  System  CSCIs.  Furthermore,  the  lack  of  complete 
software  requirements  definition  of  lower  level  software  units  such  as  Fire  Control 
and  Navigation  hampered  Lockheed  Martin’s  ability  to  perform  FCAs  and  PCAs. 

The  absence  of  FCA  and  PCA  requirements  for  internally  developed  software 
could  result  in  software  that  is  not  verified  and  validated,  leading  to  aircraft 
performance  risks. 

Verification  of  Purchased  Product  (7.4.3) 

Lockheed  Martin  received  software  from  suppliers  without  ensuring  that  known 
deficiencies  were  categorized  and  defined  in  accordance  with  its  software 
development  plan  (SDP).  For  example,  Northrop  Grumman  delivered  Integrated 
Communications,  Navigation,  and  Identification  (ICNI)  software  to  Lockheed  Martin 
with  1  major,  12  minor,  and  92  known  open  and  uncategorized  deficiencies. 
However,  neither  the  ICNI  SDP  nor  the  F-35  Air  System  SDP  defines  software  defect 
categories  or  severity  as  major  and  minor.  Lockheed  Martin  instead  categorizes 
software  deficiencies  as  Severity  1  through  5  and  does  not  allow  the  delivery  of 
flight  software  with  open  deficiencies  of  Severity  1  or  2.  Lockheed  Martin  did 
not  determine  the  system  performance  impact  and  severity  of  these  deficiencies. 

If  software  deficiencies  are  not  categorized  by  severity  as  defined  in  the 
F-35  Air  System  SDP,  then  software  could  be  released  with  open  high-severity 
defects,  which  could  result  in  performance  degradation  or  even  catastrophic 
failure  of  aircraft. 

Preventive  Action  (8.5.3) 

We  found  numerous  examples  of  software  problem  anomaly  reports  (SPARs) 
identified  with  Severity  2  problems6  that  did  not  address  root  cause  analysis  or 
effective  preventive  action.  The  root  cause  identification  and  corrective  action 
processes  are  critical  for  product  improvement  and  reliability  growth  and  the 
lack  thereof  may  also  result  in  design  and  quality  escapes  that  impact  product 
performance,  reliability,  and  maintainability. 


Joint  Strike  Fighter  Program:  Air  System  Software  Quantitative  Management  Plan,  Document  Number  2RXP00003, 
Revision  D,  defines  Severity  2  software  defects  as  "[a]  software  defect  that  adversely  affects  the  accomplishment  of  an 
operational  or  mission  essential  capability  and  no  work-around  solution  is  known." 
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Unmanaged  Performance  Specification  Risks 

Risk  Management  (7.1.2) 

JPO  and  Lockheed  Martin  jointly  identified  21  out  of  462  system-level  requirements 
contained  within  the  Joint  Strike  Fighter  (JSF)  Contract  Specification  (JCS) 
that  were  unlikely  to  be  achieved  by  the  end  of  the  SDD  phase  of  the  program. 
Examples  of  these  21  requirements  included:  Reliability  and  Maintainability, 
Maneuverability,  Payload  Requirements,  Ballistic  Vulnerability,  and  CTOL  Internal 
Gun  Accuracy.  However,  Lockheed  Martin  did  not  consider  these  21  requirements 
as  risks  and  did  not  handle  them  in  accordance  with  its  risk  management  process. 
Fulfillment  of  these  requirements  may  not  be  achieved  if  they  are  not  appropriately 
tracked  and  managed  as  risks. 

Weaknesses  in  Subcontract  Management 

Purchasing  Process  (7.4.1) 

JPO  requires  Lockheed  Martin  to  be  AS9100  compliant.  Lockheed  Martin’s  internal 
procurement  process  flows  down  AS9100  or  appropriate  quality  management 
systems  to  its  suppliers  based  on  product  complexity  and  criticality.  However, 
three  of  Lockheed  Martin’s  approved  suppliers  were  not  AS9100  certified,  as 
required  by  Lockheed  Martin’s  internal  supplier  quality  requirements.  Therefore, 
those  suppliers  might  not  have  the  necessary  quality  management  system 
processes  in  place  to  ensure  product  reliability. 

DCMA  is  the  only  JPO-delegated  authority  to  approve  minor  nonconformances. 
However,  Lockheed  Martin  procurement  requirements  allowed  minor 
nonconformances  to  be  dispositioned  and  approved  by  suppliers  who 
provide  "seller-designed”  items.  Allowing  subtier  suppliers  to  disposition 
and  approve  minor  nonconformances  impedes  the  Government’s  ability  to 
fulfill  its  responsibility  to  manage  the  risks  associated  with  the  acceptance  of 
nonconforming  products. 

Purchasing  Identification  (7.4.2) 

Lockheed  Martin’s  procurement  process  did  not  ensure  that  its  purchase  orders 
identified  the  specific  revisions  of  technical  documentation  required  to  produce, 
inspect,  and  test  the  item  being  purchased.  We  also  found  that  when  purchase 
orders  were  issued,  Lockheed  Martin  relied  on  its  suppliers  to  retrieve  the  most 
up-to-date  technical  requirements  from  their  online  technical  data  system  rather 
than  notifying  the  supplier  when  technical  data  had  changed.  These  practices 
could  result  in  products  that  are  not  compliant  with  the  applicable  technical 
requirements  in  order  to  achieve  mission  capabilities. 
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Verification  of  Purchased  Product  (7.4.3) 

Our  inspection  identified  that  Lockheed  Martin  was  not  using  its  escalation  process 
for  delinquent  supplier-generated  data  items  to  ensure  its  subcontractors  delivered 
the  required  data  items  on  time.  Hundreds  of  delinquent  supplier  data  items 
existed  with  an  average  delinquency  of  about  230  days.  Many  data  items  were 
more  than  2  years  delinquent,  with  one  item  exceeding  1,200  days.  Examples  of 
such  delinquent  data  items  included  manufacturing  plans,  acceptance  test  plans 
and  procedures,  and  diminishing  manufacturing  sources  redesign  and  technology 
refresh  plans.  Many  of  these  subcontractor  data  items  contain  information  that 
must  be  included  in  Lockheed  Martin’s  contractually  required  deliverables  to  JPO. 
Lockheed  Martin  has  an  escalation  process  to  address  delinquent  subcontract  data 
item  submittals  that  includes  withholding  funds;  however,  no  action  was  taken  to 
use  the  escalation  process  for  any  of  its  suppliers.  Manufacturing  plans,  acceptance 
test  plans  and  procedures  and  other  documentation  discussed  previously  should 
be  evaluated  by  the  appropriate  subject  matter  experts  for  adequacy.  Failure  to 
evaluate  such  deliverables  could  result  in  deficient  manufacturing  or  validation 
processes  and  ultimately  impact  product  performance  and  reliability. 

Weaknesses  in  Production  and  Assembly 

Our  inspection  of  Lockheed  Martin’s  manufacturing  and  assembly  areas  resulted 
in  the  identification  of  deficiencies  in  process  discipline,  process  planning,  process 
proofing,  tool  control  and  verification,  and  product  protection. 

Control  of  Production  and  Service  Provision  (7.5.1) 

We  identified  a  lack  of  process  discipline  in  the  manufacturing  and  assembly  area. 
In  one  instance,  multiple  fasteners  on  the  Radio  Frequency  Support  Electronics 
assembly  strut  were  not  torqued  in  accordance  with  the  applicable  specifications. 
The  manufacturing  documentation,  referred  to  as  operational  cards  and  work 
instructions,  indicated  that  both  the  operator  and  quality  inspector  had  verified 
that  the  torque  operation  was  completed.  However,  we  observed  that  the  fasteners 
were  loose  and  rotated  freely.  It  was  also  identified  that  a  mechanic  in  the  final 
assembly  area  used  sealant  on  an  arresting  gear  assembly  fasteners  after  the 
sealant’s  useful  life  had  expired. 

In  another  instance,  we  found  that  operational  card  or  planning  did  not  specify 
the  required  torque  values  for  the  mounting  of  a  cabling  connector  to  its  bracket 
on  the  frame  of  the  forward  fuselage.  Torque  requirements  for  these  fasteners 
were  prescribed  in  a  Lockheed  Martin  specification  referenced  by  the  assembly’s 
engineering  drawing.  As  a  result,  the  mechanic  installed  the  fasteners  without  the 
use  of  a  torque  wrench  and  was  not  aware  of  the  torque  requirement. 


10  |  DODIG-2015-092 


Current  F-35  Program  Quality  Management  Performance 


Lockheed  Martin  did  not  adequately  perform  process  proofing  of  the  automated 
drilling  operations  on  the  CTOL  wing.  Its  proofing  process  released  automated 
drilling  software  for  production  that  was  not  programmed  correctly  to  achieve 
full  penetration  of  the  wing  edge  in  accordance  with  the  applicable  specifications. 
Verification  operations  failed  to  identify  the  improperly  drilled  holes  before  the 
product  left  the  work  center.  These  holes  were  identified  at  the  next  work  station, 
and  the  corrective  action  taken  was  to  rework  the  holes  to  meet  the  design 
requirements  using  a  hand  drill  process. 

We  found  no  evidence  of  a  formal  process  or  management  system  to  track 
the  status  and  condition  of  critical  lift  fixtures.  As  a  result,  required  periodic 
inspection  and  load  testing  of  critical  lift  fixtures  were  not  performed  as  a  system, 
to  include:  lifting  fixture,  lifting  strap,  rings,  clevises,  and  pins.  We  also  noted  that 
a  production  tool  with  an  expired  periodic  inspection/verification  tag  was  used  in 
the  wing  assembly  area. 

We  documented  four  minor  nonconformities  related  to  Lockheed  Martin’s 
measuring  equipment  control  and  calibration  process  that  indicate  the  existence 
of  a  systemic  problem.  Examples  of  deficiencies  include:  calibration  labeling, 
maintenance  certification,  and  the  availability  of  tools  for  use  while  outside  the 
maintenance  certification  period. 

Lack  of  adequate  control  in  the  area  of  manufacturing  and  assembly  operations 
through  the  use  of  disciplined  implementation,  planning,  process  proofing,  and  tool 
control  and  verification  could  increase  defect  rates,  prevent  achievement  of  the 
planned  production  delivery  schedule,  and  result  in  negative  impacts  to  product 
performance  and  reliability. 

Preservation  of  Product  (7.5.5) 

During  our  inspection  of  the  production  areas,  we  found  one  employee  walking 
on  an  aircraft  without  the  required  shoe  covers  to  protect  aircraft  surfaces  and 
also  walking  on  surface  areas  designated  as  "no-step”  areas.  Walking  on  no-step 
areas  could  stress,  fatigue,  or  damage  aircraft  structures.  In  addition,  required 
protective  material  was  not  used  to  minimize  the  risk  of  damage  to  the  aircraft 
surfaces  by  automatic  lifting  work  stands  and  platforms. 

We  also  identified  discipline  issues  with  Foreign  Object  Debris  (FOD)  protection 
and  housekeeping.  FOD  was  detected  at  two  FOD  control  areas  in  the  final 
assembly  area.  FOD  has  continued  to  be  a  systemic  discipline  issue  since  our  last 
evaluation,  as  evidenced  by  41  Corrective  Action  Requests  (CARs)  issued  by  DCMA 
for  FOD  violations  during  2013  and  2014.  Additionally,  Lockheed  Martin  did  not 
adhere  to  housekeeping  policies  and  practices  for  cleaning  all  work  stations  and 
auto  drill  centers  of  metal  chips. 
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There  is  increased  risk  of  latent  defects  and  damage  to  the  aircraft,  if 
Lockheed  Martin  does  not  ensure  procedures  are  followed  and  product 
protection  measures  are  taken. 

Government  Quality  Assurance  Oversight  Performance 

Performance  of  DCM A 

We  evaluated  DCMA’s  oversight  of  F-35  production  and  identified  that  DCMA 
was  not  following  its  process  for  escalating  CARs.  DCMA’s  policy  is  to  escalate 
contractor  CARs  to  the  next  higher  level  when  repetitive  CARs  have  been  issued, 
when  rejected  Corrective  Action  Plans  (CAPs)  are  not  resubmitted  within  the 
10-day  requirement,  or  when  corrective  action  validation  is  rejected.  We  found 
instances  when  DCMA  failed  to  escalate  CARs  which  exemplified  each  of  these 
cases.  The  reoccurrence  of  FOD-related  problems  highlighted  the  ineffectiveness 
of  DCMA’s  corrective  action  oversight.  DCMA  issued  a  CAR  to  Lockheed  Martin  in 
response  to  FOD  control  deficiencies  documented  during  our  previous  evaluation. 
This  CAR  was  subsequently  closed  based  on  the  validation  of  completed  corrective 
actions.  During  2013  and  2014,  since  our  previous  evaluation,  DCMA  issued 
39  Level  II  CARs  for  instances  of  FOD  violations  without  escalating  them  to 
Level  III.  Failure  to  escalate  CARs  to  the  next  higher  level  when  appropriate 
undermines  the  effectiveness  of  the  corrective  action  process.  In  addition,  DCMA 
and  JPO  leadership  may  not  be  aware  of  problems  until  they  adversely  impact  the 
cost,  schedule,  and  performance  of  the  program. 

Performance  of  the  Joint  Program  Office 

As  previously  stated  in  report  section  Unmanned  Performance  Specification 
Risks,  Risk  Management  (7.1.2),  Lockheed  Martin  has  been  unable  to  achieve 
21  system-level  capability  requirements.  In  accordance  with  the  H-30,  "Future 
Specification  Changes,"  SDD  contract  clause  issued  in  May  2013,  JPO  intend  to 
change  the  21  high-risk  system-level  JCS  requirements  to  values  mutually  agreed 
upon  by  JPO  and  Lockheed  Martin  based  on  additional  testing  and  analysis  which 
is  yet  to  be  performed.  As  a  result,  delivered  and  future  aircraft  may  not  satisfy  all 
system  capability  requirements  and  it  is  unclear  to  what  extent  these  capabilities 
will  be  achieved  by  the  completion  of  the  SDD  phase. 
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As  a  part  of  this  effort,  we  evaluated  the  actions  taken  by  Lockheed  Martin,  JPO, 
and  DCMA  in  response  to  the  findings  documented  during  the  previous  evaluation 
to  determine  whether  the  root  causes  were  identified  and  the  corrective  actions 
were  effective. 

A  total  of  87  nonconformities  were  documented  against  Lockheed  Martin  during 
the  previous  evaluation;  63  were  documented  at  Fort  Worth,  Texas,  and  24  were 
documented  at  its  subcontractor  sites. 

JPO  stated  that  Lockheed  Martin  implemented  corrective  measures  to  resolve  these 
nonconformities  and  considered  82  of  them  to  be  closed.  However,  we  determined 
that  only  76  of  the  87  nonconformities  were  closed  and  we  recommended 
further  actions  for  the  11  remaining  nonconformities;  5  were  due  to  ongoing 
corrective  action  and  6  were  due  to  inadequate  corrective  action.  The  criteria  we 
used  to  determine  whether  previous  nonconformities  were  still  open  included: 
ongoing  corrective  action  efforts,  ineffective  corrective  action  to  prevent  finding 
reoccurrence,  administrative  closure  of  findings  prior  to  assessment  of  corrective 
action  effectiveness,  and  inadequate  documentation  of  corrective  actions  taken. 

To  ensure  that  all  nonconformities  requiring  further  action  from  the  previous 
evaluation  are  properly  tracked  to  closure,  we  documented  each  item  as  a  new 
nonconformity  during  this  evaluation.  We  consider  the  following  issues  still  open. 

Ongoing  Corrective  Action 
Critical  Safety  Items 

During  our  previous  evaluation,  we  documented  four  CSI  nonconformities  related 
to  the  lack  of  a  CSI  Program.  These  nonconformities  were  considered  open  by 
JPO,  DCMA,  and  Lockheed  Martin  due  to  incomplete  implementation  of  corrective 
action.  Since  our  previous  evaluation,  Lockheed  Martin  worked  with  JPO  and 
DCMA  to  develop  a  plan  to  establish  an  F-35  CSI  management  program.  Status 
and  sufficiency  of  the  plan  is  discussed  in  the  Government  Progress  on  Corrective 
Actions  section  of  this  report. 

Capability  Confirmation  Criteria 

We  also  documented  a  nonconformity  regarding  the  lack  of  capability  confirmation 
criteria  for  the  acceptance  of  LRIP  aircraft  during  our  previous  evaluation.  This 
nonconformity  was  still  considered  open  by  JPO,  DCMA,  and  Lockheed  Martin 
due  to  incomplete  implementation  of  the  corrective  action.  Since  our  previous 
evaluation,  Lockheed  Martin  worked  with  JPO  and  DCMA  to  develop  a  plan 
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to  establish  capability  confirmation  criteria.  On  LRIP  contracts,  the  program 
developed  a  Configuration  and  Capability  Description  Document  (CCDD)  to 
identify  specific  configuration  and  representative/key  capability  attributes,  such 
as  warfighting  capabilities  available  to  the  pilot  and  the  maintainer,  which  would 
be  present  in  a  respective  LRIP  lot.  However,  at  the  time  of  this  evaluation, 
Lockheed  Martin  and  JPO  were  at  a  contractual  impasse  and  were  unable  to 
implement  the  associated  corrective  actions.  JPO  and  Lockheed  Martin  had 
no  further  plan  or  path  forward  to  establish  verifiable  capability  confirmation 
criteria  for  the  acceptance  of  LRIP  aircraft.  The  lack  of  clearly  defined  capability 
confirmation  criteria  in  the  CCDD  resulted  in  acceptance  of  aircraft  whose 
capabilities  may  not  have  been  adequately  verified  and  validated. 

Ineffective  Corrective  Action 

Foreign  Object  Debris 

JPO,  DCMA,  and  Lockheed  Martin  considered  the  systemic  FOD  issues  to  be 
resolved  and  the  associated  corrective  actions  closed.  However,  we  identified 
that  the  corrective  actions  did  not  effectively  prevent  reoccurrence  of  FOD  issues. 

In  response  to  our  recommendation  from  our  previous  report,  Lockheed  Martin 
implemented  an  8-step  CAP.  Internal  Lockheed  Martin  production  assessment 
scorecard  data  for  2014  and  DCMA  Level  II  CARs  for  2013  and  2014  indicated  that 
the  steps  taken  in  the  CAP  had  not  been  effective  in  preventing  reoccurrence  of 
FOD  in  manufacturing  areas.  An  ineffective  FOD  control  program  could  lead  to 
damaged  aircraft,  degraded  performance,  catastrophic  failure,  loss  of  aircraft,  and 
loss  of  life. 

First  Article  Inspection 

First  article  inspections  are  required  to  ensure  that  manufacturing  processes 
are  validated  properly  to  reduce  variability  and  nonconformances  in  delivered 
products.  Our  previous  evaluation  at  L-3  Display  Systems  (L-3)  identified  that 
Lockheed  Martin  accepted  hardware  from  L-3  for  LRIP  lots  1  through  5  without 
the  performance  of  first  article  inspections.  JPO,  DCMA,  and  Lockheed  Martin 
considered  this  issue  to  be  resolved  and  administratively  closed  the  CAP  in 
December  2012;  however,  the  documentation  provided  with  the  CAP  stated  that  the 
sustainability  or  effectiveness  of  corrective  actions  taken  could  not  be  confirmed 
until  the  third  or  fourth  quarter  of  2013.  We  found  no  evidence  that  verification 
and  validation  efforts  were  performed  for  these  corrective  actions  by  JPO,  DCMA, 
or  Lockheed  Martin. 
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Variance  Approval  Process 

A  nonconformity  regarding  Lockheed  Martin’s  approval  of  variances  that  waived 
performance  requirements  for  the  electronics  and  display  units  was  documented 
at  L-3  during  our  previous  evaluation.  The  variances  did  not  contain  documented 
corrective  actions  that  would  prevent  the  need  for  repeated  variance  requests. 

As  part  of  the  resolution  of  this  deficiency,  Lockheed  Martin  implemented  a 

3- step  CAP  and  although  JPO,  DCMA,  and  Lockheed  Martin  considered  this  issue 
to  be  resolved  and  closed,  our  evaluation  identified  that  Lockheed  Martin’s  actions 
to  resolve  this  nonconformity  were  ineffective.  Specifically,  Lockheed  Martin’s 
procedure  and  records  indicated  that  integrated  product  team  members  were 
allowed  to  extend  the  variances  to  an  unlimited  number  of  units  and  extend 
commitment  dates  associated  with  corrective  action  completion  without 
management  approval.  Because  variance  requests  typically  impact  product 
performance — as  documented  in  this  case — the  acceptance  of  product  variances 
without  adequate  control,  oversight,  and  corrective  action  could  result  in  degraded 
aircraft  capabilities. 

Failure  Reporting  and  Corrective  Action 

During  our  previous  evaluation  we  documented  that  Lockheed  Martin  did  not 
require  UTAS  to  deliver  Failure  Reporting  and  Corrective  Action  System  (FRACAS) 
data  to  determine  the  reliability  of  the  products  delivered.  JPO,  DCMA,  and 
Lockheed  Martin  considered  this  issue  to  be  resolved  and  closed.  However, 
we  identified  that  the  corrective  actions  to  resolve  this  issue  did  not  prevent 
reoccurrence.  Since  our  last  evaluation,  Lockheed  Martin  implemented  a 

4- step  CAP  to  address  identified  deficiencies.  UTAS  delivered  the  FRACAS  data 
for  LRIP  lot  5  after  funding  issues  were  resolved;  however,  funding  constraints 
prohibited  the  delivery  of  FRACAS  data  for  LRIP  lots  6  and  7.  Lockheed  Martin 
cannot  ensure  that  the  F-35  meets  all  product  reliability  requirements  if  its 
engineering  staff  does  not  review  and  analyze  subcontractor  FRACAS  data. 

Annual  Audit  of  Software  Suppliers 

We  documented  a  nonconformity  during  our  previous  evaluation  relating 
to  Lockheed  Martin's  oversight  of  software  suppliers.  JPO,  DCMA,  and 
Lockheed  Martin  considered  this  nonconformity  to  be  resolved  and  closed; 
however,  our  evaluation  identified  that  the  corrective  action  was  inadequate. 
Lockheed  Martin  did  not  audit  its  software  suppliers  annually  in  accordance 
with  the  applicable  internal  operating  instruction.  We  evaluated  the  records 
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of  five  software  suppliers  and  noted  that  Lockheed  Martin  had  not  conducted 
required  annual  audits  for  any  of  the  suppliers  since  we  identified  this  issue  during 
our  last  evaluation.  Because  much  of  the  aircraft  capability  is  reliant  upon  robust 
software,  adequate  control  of  software  suppliers  is  necessary  to  minimize  the  risk 
of  accepting  defective  software  that  can  degrade  product  performance,  reliability, 
and  maintainability. 

Calibration  of  Measuring  Equipment 

During  our  previous  evaluation,  we  documented  that  Lockheed  Martin  granted 
temporary  extensions  of  calibration  due  dates  for  measuring  equipment  whose 
calibration  certifications  had  already  expired.  The  associated  CAP  identified  that 
the  root  cause  of  this  nonconformity  was  the  failure  of  the  equipment  owners 
to  submit  calibration  extension  requests  prior  to  expiration  with  adequate 
justification.  As  a  result,  Lockheed  Martin  removed  the  requirement  to  request 
extension  prior  to  calibration  expiration.  Based  on  this  corrective  action,  JPO, 
DCMA,  and  Lockheed  Martin  considered  the  nonconformity  resolved  and  closed; 
however,  this  could  result  in  the  increased  risk  of  using  un-calibrated  tools, 
potentially  leading  to  the  manufacture  of  nonconforming  product.  We  determined 
that  the  root  cause  analysis  and  corrective  action  should  be  reevaluated  to  prevent 
reoccurrence  of  the  problem. 

Inadequate  Objective  Evidence 

Lockheed  Martin  was  not  always  able  to  provide  objective  evidence  to  demonstrate 
that  corrective  actions  taken  to  resolve  nonconformities  identified  in  our  previous 
evaluation  were  completed.  Lockheed  Martin  uses  its  Integrated  Corrective 
Action  (ICA)  database  to  capture  internal  CARs,  CAPs,  and  objective  evidence  to 
support  CAR  closure  and  CAP  validation.  The  ICA  database  did  not  contain  all 
objective  evidence  for  numerous  CAPs  and  their  associated  DCMA  CARs  to  support 
verification  of  corrective  actions.  Closure  of  nonconformities  without  necessary 
objective  evidence  to  support  validation  of  CAP  actions  can  lead  to  reoccurrence  of 
the  nonconformities. 

Government  Progress  on  Corrective  Actions 

We  also  evaluated  DCMA’s  and  JPO’s  actions  taken  in  response  to  nonconformities 
documented  against  them  during  our  previous  evaluation  to  determine  if  root 
causes  were  identified  and  corrective  actions  were  effective. 
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Defense  Contract  Management  Agency 

During  the  previous  evaluation,  we  documented  a  total  of  six  nonconformities 
against  DCMA  offices.  These  offices  included:  DCMA  Fort  Worth,  Texas; 

DCMA  Northern  Europe,  Samlesbury,  United  Kingdom;  DCMA  Atlanta,  Georgia;  and 
DCMA  Cleveland,  Ohio.  We  identified  deficiencies  in  the  delegation  of  authority, 
minor  variance  approval,  material  review  board  actions,  issuance  and  execution  of 
letters  of  delegation,  and  delivery  of  supplier  reporting  data  between  DCMA  offices. 
Based  on  the  information  presented  by  DCMA,  we  determined  that  closure  of  these 
six  findings  was  substantiated. 

Joint  Program  Office 

Previously  Documented  Nonconformities 

We  documented  eight  nonconformities  against  JPO  during  our  previous  evaluation. 
These  nonconformities  related  to  the  CSI  Program,  documentation  of  F-35  risk 
management  activities,  F-35  diminishing  manufacturing  sources  and  materiel 
shortages  (DMSMS)  management  and  planning,  and  Block  2A  test  readiness  review 
criteria.  JPO  considered  three  nonconformities  closed;  however,  it  did  not  provide 
any  objective  evidence  for  corrective  actions  taken  in  response  to  any  of  the 
eight  nonconformities. 

Status  of  Critical  Safety  Item  Program  Implementation 

We  documented  in  our  previous  evaluation  that  JPO  and  Lockheed  Martin  did 
not  comply  with  Public  Law  108-136,  Section  802;  DFARS  Subpart  209.270; 
and  the  Joint  Service  CSI  Instruction,  "Management  of  Aviation  Critical  Safety 
Items."  Since  then,  Lockheed  Martin  created  an  extensive  CSI  list  and  DCMA 
added  the  requirement  for  government  source  inspection  for  all  CSIs  on  the  list. 
Lockheed  Martin  also  drafted  a  CSI  Program  Plan  for  submittal  to  JPO,  which 
is  expected  to  be  included  in  a  modification  to  the  SDD  contract  to  implement 
the  CSI  requirement.  The  draft  CSI  Program  Plan  erroneously  describes  the 
F-35  Program  as  "a  legacy  program  with  respect  to  CSI,"  and  unfortunately, 
implementation  of  this  plan  would  lead  to  less  than  full  compliance  with 
CSI  requirements.  At  the  time  of  our  evaluation,  JPO’s  plan  was  to  contractually 
require  implementation  of  a  CSI  program  by  February  2016.  We  determined  that 
the  draft  CSI  Program  Plan,  dated  May  21,  2014,  does  not  meet  all  the  Joint  Service 
CSI  Instruction  requirements  and  Lockheed  Martin’s  production  integrity  program 
did  not  cover  all  identified  CSIs  and  CSI  requirements. 
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During  this  evaluation,  we  identified  that  Lockheed  Martin’s  CSI  Program  Plan 
was  still  noncompliant  with  some  of  the  deficiencies  that  we  identified  in  our 
previous  report.  For  example,  the  F-35  CSI  Program  Plan  contained  a  section 
describing  the  identification  and  documentation  of  critical  characteristics.  Based 
on  that  section,  it  appeared  that  Lockheed  Martin  concluded  that  existing  quality 
and  integrity  programs  were  sufficient.  Furthermore,  the  plan  did  not  require 
CSI  critical  characteristics  to  be  defined.  It  also  did  not  ensure  CSI-specific 
training  for  personnel  performing  work  on  CSIs  or  put  the  necessary  processes  in 
place  when  operations  affect  critical  characteristics.  The  CSI  Program  Plan  still 
did  not  address  limitations  on  material  review  board  authority  for  nonconforming 
CSI  assemblies  and/or  components  or  include  scheduled  internal  and  external 
CSI  audits. 

Status  of  Independent  Quality  Organization  Establishment 
In  our  previous  report,  we  recommended  that  JPO  establish  an  independent 
quality  assurance  organization.  The  intent  of  the  recommendation  was  to 
improve  the  quality  organization’s  authority  and  independence.  JPO  did  initiate 
the  establishment  of  a  quality  organization  and  assigned  a  Quality  Lead;  however, 
the  organization  reports  to  the  Director  of  Production  rather  than  directly  to 
the  F-35  Program  Executive  Officer  (PEO).  The  F-35  Program  also  had  not  yet 
adequately  staffed  its  quality  organization  to  provide  effective  oversight,  nor  had  it 
ensured  roles  and  responsibilities  of  the  organization  were  developed  and  followed 
to  effectively  impact  the  program.  JPO’s  August  8,  2014,  response  to  our  previous 
report  stated  that  the  estimated  date  to  have  the  quality  organization  developed 
and  staffed  would  be  12  to  24  months  from  May  2014. 

Status  of  Capability  Confirmation  Criteria  Implementation 
During  this  evaluation,  we  found  that  the  capability  confirmation  criteria  necessary 
to  verify  and  validate  LRIP  aircraft  capabilities  during  aircraft  acceptance  had  not 
been  defined  in  the  CCDD,  as  documented  in  our  previous  evaluation.  As  a  result, 
the  acceptance  of  F-35  aircraft  depended  on  the  professional  judgment  of  the 
individual  JPO  subject  matter  expert  who  reviewed  the  objective  evidence  provided 
to  confirm  the  capabilities  of  the  aircraft. 

Since  our  previous  evaluation,  JPO  worked  with  Lockheed  Martin  and  DCMA  to 
develop  a  plan  to  establish  capability  confirmation  criteria  for  aircraft  acceptance; 
however,  at  the  time  of  this  evaluation,  JPO  and  Lockheed  Martin  were  at  a 
contractual  impasse  and,  as  a  result,  were  unable  to  implement  the  associated 
corrective  action.  JPO  and  Lockheed  Martin  had  no  plan  or  path  forward  to 
establish  capability  confirmation  criteria  for  the  acceptance  of  aircraft. 


18  |  DODIG-2015-092 


Summary  of  the  Evaluation 


Summary  of  the  Evaluation 

Based  on  our  evaluation,  we  determined  that  the  F-35  Program  has  made  progress; 
however,  some  challenges  still  remain. 

Although  the  quantity  of  major  nonconformities  documented  during  this  evaluation 
remained  relatively  the  same  in  comparison  to  our  previous  evaluation,  the  overall 
number  of  nonconformities  decreased  (see  Table  1).  We  further  categorized  the 
nonconformities  by  major  AS9100  section  and  compared  the  results  with  those 
of  our  previous  evaluation  (see  Figures  3  and  4).  Lockheed  Martin’s  quality 
management  system  performance  improved  in  the  Human  Resources,  Planning  of 
Product  Realization,  Design  and  Development,  and  Documentation  Requirements 
focus  areas  of  AS9100  compliance.  We  also  noted  improvement  in  the  Production 
and  Service  Provision  focus  area;  however,  the  abundance  of  major  nonconformities 
in  this  area  is  still  a  cause  for  concern.  Among  the  major  nonconformities 
documented  in  this  area,  we  observed  the  reoccurrence  of  a  systemic  FOD  problem, 
indicating  ineffective  corrective  actions.  Although  the  overall  trend  demonstrated 
improvements  in  the  majority  of  areas,  the  nonconformities  related  to  the 
Purchasing  and  Improvement  focus  areas  of  AS9100  compliance  increased. 


Table  1.  Comparison  of  Major  and  Minor  Nonconformities  Directed  to  Lockheed  Martin 
from  the  previous  report  and  this  evaluation 


Major 

Minor 

Total 

Previous  Evaluation* 

23 

40 

63 

Current  Evaluation 

24 

26 

50 

*  DoD  IG  Report  No.  DODIG-2013-140,  "Quality  Assurance  Assessment  of  the  F-35  Lightning  II  Program," 
documented  70  nonconformities  at  Lockheed  Martin,  Fort  Worth,  63  of  which  were  directed  to 
Lockheed  Martin  and  7  of  which  were  directed  to  JPO. 
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Figure  3.  Comparison  of  Major  Nonconformities  Directed  to  Lockheed  Martin 
( Previous  Evaluation  and  Current  Evaluation) 


No.  of  Nonconformities 


AS9100  Clause 
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Figure  4.  Comparison  of  Minor  Nonconformities  Directed  to  Lockheed  Martin 
(Previous  Evaluation  and  Current  Evaluation) 


DODIG-2015-092  |  21 


Summary  of  the  Evaluation 


We  also  noted  that  the  F-35  Program  made  progress  with  respect  to  its  corrective 
actions  in  response  to  nonconformities  documented  during  our  previous  evaluation. 
In  response  to  the  87  nonconformities  documented  against  Lockheed  Martin 
that  we  identified  in  our  previous  report,  Lockheed  Martin  developed  and 
implemented  corrective  action  plans  and  considered  82  of  them  resolved  and 
closed.  Lockheed  Martin  identified  and  documented  root  cause,  corrective  action 
plans,  and  provided  adequate  evidence  to  support  closure  of  the  majority  of  the 
nonconformities  documented  against  them.  However,  we  determined  that  only 
76  of  these  87  nonconformities  were  closed  and  we  recommended  further  actions 
for  the  11  remaining  nonconformities.  JPO’s  lack  of  effective  oversight  to  ensure 
timely  and  efficient  resolution  of  deficiencies  identified  in  our  previous  report  is 
demonstrated  by  the  following: 

•  limited  commitment  to  a  fully  functional  and  independent  quality 
oversight  organization, 

•  lack  of  ability  to  produce  objective  evidence  for  the  closure  of 
previous  findings  and  nonconformities, 

•  failure  to  comply  with  public  law  in  fulfilling  the 
CSI  Program  requirements, 

•  lack  of  progress  in  defining  the  acceptance  criteria  for  each  LRIP  lot, 

•  lack  of  adequate  commitment  to  ensure  that  the  program  fulfills  all 
system  level  requirements,  and 

•  lack  of  leadership  to  ensure  defect  rate  reduction  necessary  to  meet  the 
planned  full-rate  production  delivery  goals. 

In  addition,  while  DCMA  has  demonstrated  improvement,  they  are  not  following  the 
established  DCMA  CAR  escalation  process  to  manage  Lockheed  Martin’s  corrective 
action  activities  appropriately. 

Based  on  our  quality  management  system  inspection  and  evaluation  of  corrective 
actions,  we  documented  the  following  systemic  findings. 
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Finding  A 

Insufficient  Progress  Had  Been  Made  Toward 
Implementation  of  the  Critical  Safety  Item  Program 

The  F-35  Program  had  not  made  sufficient  progress  toward  full  compliance  with 
Public  Law  108-136,  Section  802,  "Quality  control  in  procurement  of  aviation 
CSIs  and  related  services,”  and  the  Joint  Service  CSI  Instruction,  "Management 
of  Aviation  Critical  Safety  Items,”  by  February  2016.  This  was  caused  by  the 
lack  of  CSI  requirements  on  the  F-35  contracts,  as  required  by  public  law  and 
the  Joint  Service  CSI  Instruction.  This  lack  of  compliance  with  CSI  public  law 
and  implementing  instructions  may  impose  an  increased  safety  of  flight  risk  on 
delivered  aircraft. 


Discussion 

As  identified  in  our  previous  report  JPO  had  not  flowed  down  a  contractual 
requirement  to  Lockheed  Martin  to  implement  a  CSI  Program  in  accordance 
with  Public  Law  108-136,  Section  802  and  the  Joint  Service  CSI  Instruction. 

The  CSI  public  law  has  been  imposed  since  2003. 

JPO  and  Lockheed  Martin  have  taken  initial  steps  to  establish  a  CSI  Program; 
however,  the  program  was  still  not  fully  compliant  with  applicable  laws  and 
regulations.  JPO  was  planning  to  contractually  require  implementation  of  a 
CSI  Program  by  February  2016.  However,  the  current  draft  of  the  CSI  Program 
Plan  prepared  by  Lockheed  Martin  did  not  sufficiently  address  requirements 
described  in  the  Joint  Service  CSI  Instruction  including  the  need  to  define  critical 
characteristics  for  all  CSIs  and  the  proper  limitations  and  delegations  for  the 
material  review  board  authority  for  nonconforming  CSI  critical  characteristics. 
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Management  Comments  on  the  Finding  and 
Our  Response 

Summaries  of  management  comments  on  the  finding  and  our  response  are  in 
Appendix  C. 

Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  A 

We  recommend  that  the  F-35  Joint  Program  Office: 

1.  Ensure  that  the  F-35  CSI  Program  is  compliant  with  Public  Law  108-136, 
Section  802,  "Quality  control  in  procurement  of  aviation  CSls  and  related 
services,"  and  the  Joint  Service  CSI  Instruction,  "Management  of  Aviation 
Critical  Safety  Items." 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  agreed  and  stated: 

•  Contract  actions  for  both  SDD  and  LRIP  that  address  the  Joint  Service 
CSI  Instruction,  "Management  of  Aviation  CSIs"  are  in  work  or  in  place  to 
implement  the  full  F-35  CSI  Program 

o  SDD  RFP  for  CSI  NRE  [non-recurring  engineering]  efforts  to  initiate 
F-35  CSI  Program  released  Oct  2013  and  resulted  in  a  LM  delivered 
Draft  CSI  Program  Plan  coordinated  with  F-35  JPO  for  approval  of 
the  CSI  implementation  approach  and  go  ahead  for  delivery  of  the 
SDD  proposal  expected;  go  ahead  date  for  proposal  development 
expected  2QFY15 

•  After  the  SDD  CSI  NRE  efforts  are  completed,  the  full  scope 
of  the  F-35  CSI  program  will  be  included  in  future  production 
and  sustainment  contracts. 

o  LRIP  Production  contracts  (LRIP  1-8)  currently  cover  explicit 
CSI  tasks  for  'limited'  compliance  for  several  CSI  'gaps'  from  the 
Joint  Service  CSI  Instruction,  "Management  of  Aviation  CSIs." 

•  Maintain  /  update  the  DOE  [Director  of  Engineering] 
approved  CSI  list  for  configuration  control 

•  Use  existing  recurring  processes:  Material  Review  (MR), 
Variance,  Item  Acceptance,  New  Suppliers,  Process  Metrics, 
and  Minor  &  Major  Change  processes  for  CSIs 

•  Provide  CSI  notification  to  affected  suppliers  via 
contracts  letter 

•  Provide  notification  if  any  identified  non-conformances 
results  in  a  safety  impact 
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•  Report  MR  actions  &  QARs  [Quality  Assurance  Reports] 
associated  with  CSI  once  per  month 

•  Use  existing  disposal  processes  to  destroy  scrapped  CSIs 
that  render  part  nonfunctional 

•  DCMA-LM  FW  [Fort  Worth]  Integrated  CSI  Surveillance  Strategy  developed 
Jan  2014  and  covers  CSI  Delegations  to  ensure  appropriate  DCMA  process 
surveillance  of  CSI  parts,  and  imposed  Government  Source  Inspection  (GSI) 
on  all  parts  that  have  been  identified  as  CSIs  per  the  Joint  Service 
CSI  Instruction. 

Our  Response 

The  comments  from  JPO  partially  meet  the  intent  of  our  recommendation. 

Based  on  JPO's  response,  the  F-35  program  appears  to  be  making  progress  in 
establishing  a  CSI  program.  However,  the  comments  do  not  include  a  timeline  for 
the  implementation  of  the  “full"  F-35  CSI  Program.  We  request  that  JPO  provide 
a  timeline  for  the  implementation  of  the  "full"  F-35  CSI  Program  and  update  the 
current  draft  CSI  Program  to  fully  comply  with  applicable  laws  and  regulations. 

2.  Conduct  periodic  CSI  Program  evaluations  of  Lockheed  Martin  and  its 
suppliers  to  ensure  compliance  with  public  law  and  the  Joint  Service 
CSI  Instruction. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  agreed  and  stated: 

F-35  JPO  &  LM  [Lockheed  Martin]  conducted  a  CSI  TIM  [Technical  Interchange 
Meeting]  in  Nov  2014  following  completion  of  the  DoD  IG  Quality  Assurance 
&  Corrective  Action  Evaluation  to  address  DoD  IG  "Opportunity  For 
Improvement"  comments  issued  for  CSIs  and  CSI  Program  Plan. 

F-35  JPO,  DCMA  &  LM  are  conducting  bi-weekly  telecoms  and  periodic 
TIMs  on-site  at  LM  until  SDD  contract  agreement  is  reached  to  implement 
the  full  scope  CSI  Program  infrastructure  and  processes. 

After  F-35  full  scope  CSI  implementation  is  underway,  the  F-35  JPO  will 
transition  to  a  monitoring  phase  with  periodic  reviews,  which  will  leverage  or 
combine  with  other  JPO/DCMA  activities  (joint  audits,  Production  Readiness 
Reviews,  quality  reviews,  metrics,  etcetera]. 


Our  Response 

The  comments  from  JPO  meet  the  intent  of  our  recommendation.  No  further 
comments  are  required. 
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Finding  B 

Oversight  of  Product  Development  and  Realization  of 
Requirements  Was  Inadequate 

JPO  was  not  adequately  managing  the  F-35  Program  to  ensure  that  all  system-level 
requirements  and  capabilities  defined  in  the  JSF  JCS  were  realized  and  verified. 

As  a  result,  delivered  and  future  aircraft  may  not  satisfy  all  system  capability 
requirements  and  it  is  unclear  to  what  extent  these  capabilities  will  be  achieved 
by  the  completion  of  the  SDD  phase. 


Discussion 

The  JCS  document  identifies  more  than  400  system-level  requirements  that 
define  performance  characteristics.  JPO  and  Lockheed  Martin  jointly  identified 
21  system-level  requirements  contained  within  the  JCS  that  were  unlikely  to 
be  achieved  by  the  end  of  the  SDD  phase  of  the  program.  However,  JPO  and 
Lockheed  Martin  did  not  consider  these  21  requirements  as  risks  and  did  not 
handle  them  in  accordance  with  their  risk  management  process. 

In  addition,  JPO  still  has  not  defined  objective  and  measureable  aircraft  acceptance 
criteria  for  each  LRIP  lot.  The  confirmation  criteria  defined  in  the  CCDD  did  not 
include  clearly  measurable  and  verifiable  acceptance  criteria  that  were  clearly 
traceable  back  to  the  JCS.  This  was  documented  in  our  previous  evaluation  and 
continues  to  be  a  challenge. 

Management  Comments  on  the  Finding 
and  Our  Response 

Summaries  of  management  comments  on  the  finding  and  our  response  are  in 
Appendix  C. 

Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  B 

We  recommend  that  the  F-35  Joint  Program  Office: 

1.  Clearly  define  contractual  criteria  for  the  acceptance  of  all  future  and 
fielded  aircraft  to  ensure  that  aircraft  capabilities  are  verified. 
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F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  agreed  and  stated: 

F-35  JPO  is  progressively  refining  acceptance  criteria  with  each  LRIP  lot. 

The  use  of  CCDDs  will  continue  to  be  used  for  LRIP  7  and  LRIP  8.  These  CCDD 
items  require  positive  verification  activity  to  occur,  often  with  dedicated  flight 
test  evaluations;  and  they  provide  a  means  to  ensure  the  weapons  system 
meets  warfighter  expectation  for  that  production  lot.  However,  the  use  of 
these  documents  does  not  negate  the  responsibility  to  document  any  known 
JCS  non-compliances  at  the  time  of  aircraft  acceptance.  This  will  continue  to 
occur  as  it  has  occurred  in  the  past.  As  SDD  progresses  F-35  JPO  does  expect 
more  qualification  evidence  to  be  available  which  will  allow  for  more  robust 
documentation  of  JCS  non-compliance  at  the  time  of  aircraft  acceptance.  This 
will  be  done  in  addition  to  requiring  positive  verification  for  all  CCDD  items. 

Beginning  with  LRIP  9,  the  next  lot  to  be  negotiated,  F-35  JPO  will  require  full 
JCS  compliance.  F-35  JPO  will  assess  LRIP  9+  product  acceptance  against  the 
SDD  specification.  For  future  full-rate  production  contracts,  there  is  planned  a 
separate  production  specification  that  will  leverage  the  final  SDD  specification. 


Our  Response 

The  comments  from  JPO  do  not  meet  the  intent  of  our  recommendation.  JPO  has 
not  identified  the  actions  that  it  will  take  for  verification  and  validation  of  aircraft 
already  delivered  or  still  in  production  for  LRIP  lots  1  through  8.  JPO  needs  to 
ensure  all  aircraft  delivered  are  tested  and  verified  using  clear  and  measurable 
capability  criteria.  We  request  that  JPO  provide  further  comments  in  response  to 
the  final  report. 

2.  Ensure  that  all  21  system-level  requirements  that  the  F-35  Program  has 
acknowledged  may  not  be  met,  in  addition  to  the  risks  associated  with 
the  failure  to  meet  these  requirements,  are  documented,  tracked,  and 
mitigated  using  the  established  risk  management  process. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  partially  agreed  and  stated: 

F-35  JPO  sees  merit  in  tracking  to  closure  the  contractually  defined 
21  system  level  requirements  that  may  not  be  fully  met.  F-35  JPO  agrees 
the  high-level  visibility  of  a  formal  tracking  Action  Item  to  periodically 
status  these  requirements  to  closure  will  ensure  resolution  of  them  in  an 
expeditious  manner. 

However,  F-35  JPO  DISAGREES  with  the  recommendation  to  track  these 
21  system  level  requirements  through  the  risk  management  process.  These 
21  requirements  are  not  risks  -  they  are  a  subset  of  requirements  where  the 
program  has  jointly  acknowledged  that  the  level  of  specified  performance 
will  not  be  met.  Conversely,  the  purpose  of  risk  management  is  to  highlight 
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and  take  proactive  actions  to  avert  a  potentially  bad  outcome,  such  as  a 
specification  miss.  Within  the  F-35  risk  management  process,  risks  are 
managed  via  a  series  of  funded  mitigation  steps  and  technical  gates  to 
eliminate  or  reduce  the  likelihood  and/or  consequence  of  each  risk.  With 
these  21  requirements,  F-35  JPO  has  already  acknowledged  the  specification 
misses  and  has  a  contractual  agreement  to  spend  no  more  money  to  improve 
the  design  -  using  the  cost  as  an  independent  variable  principle  for  the 
requirements.  The  only  remaining  program  resources  associated  with  these 
21  requirements  is  for  gathering  qualification  evidence  to  verify  the  capability 
of  the  design  as  is.  When  qualifications  are  complete,  F-35  JPO  will  reset 
the  specification  requirements  to  the  achieved  performance  and  negotiate 
consideration  for  each  of  the  shortfalls. 

A  closure  tracking  /  status  Action  Item  for  these  21  system  level  requirements 
would  regularly  assess  progress  toward  reconciliation  and  closure  of  the 
specification  to  realized  performance:  qualification  actions,  coordination  of 
verification  evidence,  draft  specification  change  request  engineering  review, 
Configuration  Steering  Board  approval,  Change  Control  Board  approval 
and  contract  negotiations  /  contract  modifications.  However,  for  these 
21  system  level  requirements,  neither  risk  mitigations  nor  design  changes  are 
funded  within  the  remaining  scope  of  the  System  Design  and  Development 
contract.  As  an  aside,  several  of  the  21  system  level  requirements  are 
separately  tracked  as  Program  Technical  Performance  Measures. 


Our  Response 

The  comments  from  JPO  do  not  meet  the  intent  of  our  recommendation.  Although 
JPO  stated  that  the  requirements  will  not  be  met,  the  undetermined  extent  to  which 
the  21  system-level  requirements  will  not  be  met  poses  a  risk  to  the  F-35  Program. 
These  21  system-level  requirements  should  be  tracked  as  program-level  risks 
either  until  the  requirements  have  been  satisfied  or  relief  has  been  granted  by  the 
appropriate  authority.  We  request  that  JPO  provide  further  comments  in  response 
to  the  final  report. 


28  |  DODIG-2015-092 


Findings 


Finding  C 

The  Quality  Assurance  Organization  Was  Not 
Independent  and  Not  Adequately  Staffed 

JPO  had  not  defined  its  quality  assurance  organization’s  roles  and  responsibilities 
and  had  not  provided  adequate  staffing  to  perform  effective  oversight  of  the 
F-35  Program.  Also,  the  quality  assurance  organization  reports  to  the  Director 
of  Production  rather  than  directly  to  the  F-35  PEO.  The  JPO  quality  assurance 
function  did  not  have  the  necessary  independence  to  implement  the  quality 
management  mission  for  development,  production,  and  sustainment  of  the  system. 
A  quality  assurance  organization  that  lacks  independence  or  the  inherent  authority 
to  enforce  quality  management  requirements  may  not  effectively  mitigate  cost, 
schedule,  and  performance  risks  to  the  F-35  Program. 


Discussion 

Our  previous  evaluation  recommended  that  JPO  establish  an  independent  quality 
assurance  organization.  JPO  initiated  the  establishment  of  a  quality  organization, 
assigned  a  Quality  Lead,  and  plans  to  further  develop  and  staff  the  organization. 
But,  the  organization  reports  to  the  Director  of  Production  rather  than  directly 
to  the  F-35  PEO.  Because  the  desire  to  meet  production  schedules  can  often 
prevent  adequate  or  sufficient  root  cause  analysis  and  corrective  action,  a  quality 
assurance  organization  that  is  independent  and  reports  to  the  PEO  ensures  greater 
influence  in  program  decisions.  The  current  organizational  structure  may  limit 
the  quality  assurance  organization’s  ability  to  objectively  report  program  quality 
management  performance. 

Management  Comments  on  the  Finding 
and  Our  Response 

Summaries  of  management  comments  on  the  finding  and  our  response  are  in 
Appendix  C. 
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Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  C 

We  recommend  that  the  F-35  Joint  Program  Office  realign  the  quality  assurance 
organization  to  report  directly  to  the  PEO,  define  the  organization's  roles  and 
responsibilities,  and  staff  the  organization  appropriately. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  agreed  and  stated: 

Efforts  to  establish  an  independent  quality  organization  are  on  track. 

As  documented  in  F-35  JPO's  August  2014  response  and  reiterated  during 
presentations  and  discussions  with  DODIG  team  members,  a  quality 
organization  will  be  in  place  12-24  months  from  the  May  2014  decision. 


Our  Response 

The  comments  from  JPO  do  not  meet  the  intent  of  our  recommendation. 

JPO  has  proposed  inconsistent  approaches.  JPO’s  response  letter  addressing 
the  recommendation  in  our  previous  evaluation  to  establish  an  independent 
quality  organization  stated,  " [t]he  F-35  JPO  will  establish  an  independent  quality 
organization  reporting  to  the  Program  Executive  Officer.”  However,  during  this 
evaluation,  JPO  presented  a  quality  organization  that  has  a  direct  reporting 
line  to  the  Director  of  Production,  which  we  do  not  consider  to  be  independent. 

In  addition,  it  is  not  clear  from  JPO’s  response  whether  the  quality  organization 
will  be  fully  functional  12  to  24  months  from  the  referenced  May  2014  decision. 
Timely  establishment  of  the  quality  organization  is  essential  to  provide  necessary 
oversight,  monitor  the  program  performance,  and  take  proactive  measures  to 
prevent  significant  risks  to  the  program.  We  recommend  that  JPO  provide  an 
update  to  the  schedule  and  milestones  associated  with  the  development  of  the 
quality  organization,  including  a  current  and  proposed  quality  organization 
structure.  We  request  that  JPO  provide  further  comments  in  response  to  the 
final  report. 
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Finding  D 

Reduction  of  the  Assembly  Defect  Rate 
Was  Inadequate 

JPO  and  DCMA  management  were  not  committed  to  ensuring  that  assembly  defect 
rate  objectives  and  goals  were  realized.  Lack  of  ability  to  reduce  defects  and  the 
scrap,  rework,  and  repair  rate  in  accordance  with  defined  program  objectives  will 
negatively  impact  program  cost  and  schedule. 


Discussion 

JPO  was  not  ensuring  that  Lockheed  Martin  was  fulfilling  its  commitment  to 
a  90-percent  assembly  defect  rate  reduction  per  aircraft  to  meet  the  full-rate 
production  goal  of  completing  one  aircraft  per  day.  We  were  not  able  to  determine 
Lockheed  Martin’s  progress  in  meeting  its  commitment  to  a  90-percent  reduction  in 
the  assembly  defect  rate  because  it  was  not  tracking  its  progress  using  the  correct 
data  for  SDD  or  LRIP  lot  deliveries.  In  accordance  with  conflicting  information 
contained  in  the  quality  management  plan,  Lockheed  Martin  was  tracking  the 
number  of  nonconforming  material  records  rather  than  the  number  of  assembly 
defects.  Lockheed  Martin  demonstrated  a  23-percent  reduction  in  nonconforming 
material  records  compared  to  the  60-percent  defect  rate  reduction  that  should 
have  been  realized  after  LRIP  lot  5.  In  addition,  JPO  and  DCMA  did  not  ensure  that 
corrective  actions  for  product  nonconformances  were  adequately  implemented  to 
reduce  the  assembly  defect  rate.  At  the  time  of  our  evaluation,  we  were  not  aware 
of  any  initiatives  or  value  engineering  change  proposals  specifically  aimed  at  the 
reduction  of  the  assembly  defect  rate  in  accordance  with  the  program  quality 
management  plan  goals. 

Management  Comments  on  the  Finding 
and  Our  Response 

Summaries  of  management  comments  on  the  finding  and  our  response  are  in 
Appendix  C. 
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Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  D 

We  recommend  that  the  F-35  Joint  Program  Office: 

1.  Ensure  that  Lockheed  Martin  implements  quality  improvement  initiatives 
to  reduce  the  assembly  defect  rate  to  meet  full-rate  production  goals. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  agreed  and  stated: 

As  presented,  Lockheed  Martin  has  initiated  several  initiatives  to  reduce 
nonconformances  of  all  types  both  within  their  facilities  and  throughout  their 
supply  chains.  The  F-35  JPO  is  committed  to  ensuring  quality  objectives 
and  goals  are  realized,  therefore  monitors  progress  closely  with  monthly 
and  quarterly  quality  metrics  reviews  and  Production  Readiness  Reviews. 
Additionally  F-35  JPO  production  operations  and  quality  subject  matter 
experts  attend  Systems  Engineering  Technical  Reviews  to  assure  designs 
are  producible  and  encompass  quality  objectives.  Additional  improvement 
incentives  are  being  realized  through  the  program's  Blueprint  for  Affordability 
which  provides  a  cost  share  between  the  contractor  and  government  to  fund 
the  improvements,  while  not  solely  dedicated  to  quality  and  producibility 
improvements  the  majority  of  projects  achieve  savings  through  defect 
reduction  by  focusing  on  producibility.  F-35  JSPO  [Joint  Strike  Fighter 
Program  Office]  is  satisfied  with  the  methodology  LM  is  using  to  monitor  SRR 
in  lieu  of  a  ratio  to  production  hours.  Additional  metrics  are  being  evaluated 
by  F-35  JPO,  DCMA  and  the  contractors  independently  and  jointly  to  continue 
driving  discovery  of  nonconformances  at  their  source  (eliminate  escapes] 
as  well  as  facilitating  correction.  The  mutually  agreed  to  quality  goals  are 
part  of  the  Joint  Program  Commitments,  which  the  F-35  JPO  may  consider 
implementing  in  contracts,  though  they  are  not  currently. 


Our  Response 

The  comments  from  JPO  partially  meet  the  intent  of  our  recommendation; 
however,  we  disagree  with  JPO’s  position  that  Lockheed  Martin’s  approach  satisfies 
the  F-35  program’s  quality  objectives  as  defined  by  the  F-35  Program  Quality 
Management  Plan.  According  to  the  quality  management  plan,  analysis  of  assembly 
defects  and  Quality  Assurance  Reports  per  aircraft  is  necessary  to  determine  the 
progress  in  meeting  the  objectives.  JPO  needs  to  work  with  Lockheed  Martin  to 
identify  quality  metricfs)  that  could  be  used  contractually  to  ensure  that  quality 
issues  per  aircraft  are  reduced  to  meet  program  goals.  We  request  that  JPO 
provide  further  comments  in  response  to  the  final  report. 
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2.  Coordinate  with  DCMA  to  implement  an  effective  root  cause  analysis  and 
corrective  action  process  in  order  to  reduce  assembly  defect  rate. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  partially  agreed  and  stated: 

There  are  effective  Corrective  Action  processes  already  in  place  within 
Lockheed  Martin  and  the  supply  chain.  DCMA  and  F-35  JPO  quality 
organizations  focus  heavily  on  continually  improving  these  systems  and 
process  to  assure  cause  analysis  leads  to  true  root  cause  and  does  not  stop 
with  direct  cause.  This  includes  areas  outside  production,  including  work 
instructions,  tooling,  design,  facilities,  human  resources  and  even  within  the 
government  should  government  furnished  equipment  property  or  direction 
be  a  contributor.  Lockheed  Martin  has  also  implemented  internal  corrective 
action  processes  for  non-shop  floor  continual  improvement  opportunities. 
While  F-35  JPO  agrees  effective  root  cause  and  corrective  action  are  essential 
to  improvement,  beyond  continuous  high  priority  focus  through  Corrective 
Action  Board  meetings,  metrics  reviews,  DCMA  issuance  of  requests  for 
corrective  action  no  additional  changes  are  necessary  at  this  time. 


Our  Response 

The  comments  from  JPO  do  not  meet  the  intent  of  our  recommendation.  Our  intent 
was  for  JPO  to  take  actions  to  identify  and  correct  the  cause  of  the  program's 
inability  to  meet  established  quality  goals  and  objectives  for  full-rate  production  as 
stated  in  the  F-35  Program  Quality  Management  Plan.  JPO  should  work  with  DCMA 
and  Lockheed  Martin  to  reduce  the  number  of  assembly  defects  per  aircraft  though 
quality  improvement  initiatives.  We  request  that  JPO  provide  further  comments  in 
response  to  the  final  report. 
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Finding  E 

Corrective  Action  Request  Escalation  Was  Inadequate 

DCMA  Fort  Worth  was  not  escalating  unresolved  CARs  to  the  next  higher  level  as 
required  by  the  applicable  DCMA  instruction.  If  DCMA  does  not  escalate  CARs 
as  required,  deficiencies  may  continue  to  occur  and  DCMA  and  JPO  leadership 
may  not  be  aware  of  problems  until  they  adversely  impact  the  cost,  schedule,  and 
performance  of  the  program. 


Discussion 

DCMA  was  not  following  its  policy  to  escalate  CARs  to  the  next  higher  level 
when  repetitive  Level  II  CARs  have  been  issued,  when  CAPs  were  rejected 
and  not  resubmitted  within  the  10-day  requirement,  or  when  the  corrective 
action  validation  is  rejected.  We  identified  instances  when  multiple  CARs  were 
documented  for  the  same  violation,  but  none  were  escalated  to  the  next  level. 

We  also  noted  instances  when  CAPs  were  rejected  and  were  not  resubmitted 
within  the  10-day  requirement,  but  did  not  result  in  DCMA  escalating  the  CAR  to 
the  next  level. 

Management  Comments  on  the  Finding 
and  Our  Response 

Summaries  of  management  comments  on  the  finding  and  our  response  are  in 
Appendix  C. 
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Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  E 

We  recommend  that  the  Defense  Contract  Management  Agency: 

1.  Review  all  unresolved  CARs  and  escalate  those  that  meet  the  criteria 
established  in  DCMA  policies  and  instructions. 

DCMA  Comments 

DCMA  agreed  and  stated: 

DCMA  agrees  we  did  not  execute  to  DCMA  Instruction  #1201  as  interpreted 
by  the  DoD  IG  team.  We  agree  that  DCMA  Instruction  1201  requires 
clarification.  It  is  currently  under  revision  to  address  clarification  to  the  CAR 
elevation  process.  DCMA  Headquarters  will  provide  an  updated  policy  with 
an  estimated  completion  date  of  early  2015.  Upon  policy  update  approval, 
DCMA  LMFW  [Lockheed  Martin,  Fort  Worth]  will  review  all  open  CARS  to 
determine  if  any  warrant  elevation  to  the  next  level  as  recommended. 


Our  Response 

The  comments  from  DCMA  meet  the  intent  of  our  recommendation.  No  further 
comments  are  required. 

2.  Assess  all  CARs  that  were  not  properly  elevated  and  assess  any  impact  on 
the  product. 

DCMA  Comments 

DCMA  agreed  and  stated: 

DCMA  LMFW  will  assess  all  open  CARs  based  on  new  policy  guidance  to 
identify,  analyze,  and  assess  any  impacts  to  F-35  product. 


Our  Response 

The  comments  from  DCMA  meet  the  intent  of  our  recommendation.  No  further 
comments  are  required. 
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Finding  F 

Software  Quality  Management  Was  Insufficient 

JPO  and  Lockheed  Martin  were  not  ensuring  software  quality  management 
processes  such  as  configuration  management,  failure  reporting  and  corrective 
action,  nonconformance  root  cause  and  analysis,  and  training  were  performed 
sufficiently  to  prevent  delayed  mission  capability,  software  rework,  and  possibly 
additional  program  costs.  In  addition,  undiscovered  software  defects  could  result 
in  performance  degradation  or  even  catastrophic  failure  of  aircraft,  subsequently 
resulting  in  loss  of  aircraft  and  death. 


Discussion 

Lockheed  Martin's  software  development  methodology  did  not  facilitate  the 
performance  of  physical  and  functional  configuration  audits.  The  majority  of 
new  aircraft  capability  is  based  on  additional  software  builds;  therefore,  it  is 
necessary  to  perform  functional  and  physical  configuration  audits.  These  types 
of  audits  ensure  that  CSCIs  exist  and  are  configured,  tested,  and  documented  in 
accordance  with  software  specifications  and  requirements  to  mitigate  aircraft 
performance  and  reliability  risks.  In  addition,  deficiencies  in  Lockheed  Martin's 
software  development  training  program  may  lead  to  knowledge  gaps  which  could 
result  in  software  development  process  failures  and  software  defects.  Additionally, 
the  classification  of  the  severity  of  software  deficiencies  identifies  the  level  of 
verification  and  corrective  action  required.  Lockheed  Martin's  failure  to  ensure 
software  subcontractors,  such  as  Northrop  Grumman,  were  categorizing  software 
deficiencies  in  accordance  with  the  F-35  Air  System  Software  Development  Plan 
does  not  provide  confidence  in  delivered  software.  Allowing  open  deficiencies, 
especially  those  whose  severities  are  not  characterized  and  understood,  can  result 
in  performance  degradation  or  catastrophic  failure  during  operation.  JPO  needs  to 
ensure  that  rigorous  software  processes  exist  at  Lockheed  Martin. 
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Management  Comments  on  the  Finding 
and  Our  Response 

Summaries  of  management  comments  on  the  finding  and  our  response  are  in 
Appendix  C. 

Recommendation,  Management  Comments, 
and  Our  Response 

Recommendation  F 

We  recommend  that  the  F-35  Joint  Program  Office  work  with  Lockheed  Martin 
to  ensure  software  quality  management  systems  are  improved;  metrics 
should  be  reported  on  a  periodic  basis  (for  example,  monthly)  to  evaluate 
process  improvement. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  agreed  and  stated: 

The  F-35  JPO  has  been  and  will  continue  to  work  with  Lockheed  Martin 
to  ensure  the  adequacy  and  improvement  of  software  quality  management 
processes  and  practices.  Metrics  reflecting  software  quality  and  quality 
improvement  objectives  have  been  developed  and  are  being  reported  and 
assessed  at  both  the  vehicle  systems  and  mission  systems  team  levels  on  a 
monthly  basis,  as  well  as  at  the  air  vehicle  level  and  production  quality  on  a 
monthly  basis. 


Our  Response 

The  comments  from  JPO  partially  meet  the  intent  of  our  recommendation.  JPO  does 
not  give  the  impression  that  any  changes  and/or  improvement  will  be  made  to  the 
software  quality  management  process  as  a  result  of  the  finding.  We  request  that 
JPO,  in  response  to  the  final  report,  provide  additional  information  regarding  the 
types  of  improvements  that  have  been  implemented  to  show  that  critical  software 
verification  processes  are  implemented  at  various  levels  to  ensure  software 
product  integrity. 
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Finding  G 

Subcontract  Management  Procedures  Were 
Still  Inadequate 

JPO,  DCMA,  and  Lockheed  Martin  were  not  ensuring  that  contract  requirements 
were  adequately  flowed  down,  contract  deliverables  were  evaluated,  and  minor 
nonconformances  were  approved  by  the  proper  authority.  Lack  of  sufficient 
subcontract  management  activities  may  result  in  the  acceptance  of  products  that 
do  not  meet  intended  performance  and  reliability  requirements. 


Discussion 

DCMA  and  Lockheed  Martin  were  not  ensuring  that  the  purchase  orders  identified 
the  specific  revisions  of  the  technical  requirement  documentation  necessary  for 
the  production,  testing,  and  inspection  of  each  item  being  purchased  and  changes 
to  technical  requirements  were  contractually  documented.  They  also  did  not 
ensure  that  contractually  required  supplier-generated  data  items  were  delivered  by 
subcontractors  when  required.  We  found  no  evidence  of  any  appropriate  actions 
taken  when  subcontractors  did  not  comply  with  these  requirements.  In  addition, 
JPO  and  Lockheed  Martin  were  not  ensuring  that  disposition  and  approval  of  minor 
nonconformances  were  approved  only  by  DCMA,  who  has  the  delegated  authority 
for  approval  of  minor  nonconformances. 

Recommendations,  Management  Comments, 
and  Our  Response 

Recommendation  G 

1.  We  recommend  that  the  F-35  Joint  Program  Office  ensure  that  all  minor 
nonconformances  are  evaluated  and  approved  only  by  DCMA. 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  partially  agreed  and  stated: 

F-35  JPO  will  work  with  DCMA  to  assure  compliance  with  regulation  and 
direction.  As  part  of  an  existing  effort  to  validate  requirements  flow  down 
accuracy,  minor  nonconformance  approval  will  be  included.  Scrap  and 
rework-to-design-definition  (so  called  rework  to  blue-print]  dispositions 
ultimately  result  in  conforming  product  being  presented  to  the  government 
for  acceptance  and  alleviate  the  requirement  of  further  DCMA  review.  DCMA 
has  and  will  continue  to  intercede  in  repetitive  scrap  and  rework.  In  the 
case  of  repair  or  use-as-is  disposition,  DCMA  reviews  these  in  accordance 
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with  regulation  and  instruction  as  it  results  in  nonconforming  product  being 
tendered  for  acceptance.  The  F-35  JPO  will  work  with  DCMA  to  assure 
nonconformance  evaluations  are  properly  reviewed  in  accordance  with 
requirements  and  action  plans  put  in  place  by  March  31,  2015. 


Our  Response 

The  comments  from  JPO  meet  the  intent  of  the  recommendation.  No  further 
comments  are  required. 

2.  We  recommend  that  the  Defense  Contract  Management  Agency  ensure 
that  Lockheed  Martin  flows  down  the  appropriate  technical  requirements 
to  its  subcontractors,  and  receives  and  evaluates  contract  deliverables 
within  the  required  time  frames. 

DCMA  Comments 

DCMA  agreed  and  stated: 

DCMA  LM  will  assess  current  functional  surveillance  strategies  to  determine 
system/process  surveillance  gaps  regarding  Subcontract  Management 
Procedures.  DCMA  LM  surveillance  strategies  will  be  revised  as  necessary 
to  ensure  LM  processes  are  adequate  to  provide  appropriate  flow  down 
of  requirements  and  execution  of  those  requirements  are  achieved  by 
their  subcontractors. 


Our  Response 

The  comments  from  DCMA  meet  the  intent  of  the  recommendation.  No  further 
comments  are  required. 
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Appendix  A 

Scope  and  Methodology 

We  conducted  this  evaluation  from  August  2014  through  December  2014  in 
accordance  with  the  Council  of  the  Inspectors  General  on  Integrity  and  Efficiency, 
"Quality  Standards  for  Inspection  and  Evaluation."  Those  standards  require  that 
we  plan  and  perform  the  evaluation  to  obtain  sufficient,  appropriate  evidence  to 
provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  evaluation 
objectives.  We  believe  that  the  evidence  obtained  provides  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  evaluation  objectives. 

The  scope  of  this  evaluation  was  to  determine  the  F-35  Program  conformity 
to  the  contractually  required  AS9100,  "Quality  Management  Systems  - 
Requirements  for  Aviation,  Space  and  Defense  Organizations,"  standard  and 
whether  the  F-35  Joint  Program  Office  (JPO),  Defense  Contract  Management 
Agency  (DCMA),  and  the  prime  contractor — Lockheed  Martin  Aeronautics 
Company  took  appropriate  actions  on  findings  and  recommendations  identified 
in  the  DoD  IG  Report  No.  DODIG-2013-140,  "Quality  Assurance  Assessment  of  the 
F-35  Lightning  II  Program,"  September  30,  2013.  We  performed  a  two-week  AS9100 
inspection  at  Lockheed  Martin  Aeronautics  Company,  Fort  Worth,  Texas,  from 
September  22,  2014  to  October  3,  2014. 

The  AS9100C  standard  breaks  down  quality  assurance  requirements  into 
five  major  clauses: 

•  Quality  Management  System, 

•  Management  Responsibility, 

•  Resource  Management, 

•  Product  Realization,  and 

•  Measurement,  Analysis,  and  Improvement. 

The  first  three  major  AS9100  clauses,  Quality  Management  System,  Management 
Responsibility,  and  Resource  Management,  require  the  organization  to  have 
a  quality  assurance  management  organization  that  has  all  the  resources  and 
authority  to  affect  the  end-item  quality  of  the  product.  In  addition,  it  requires 
the  organization  to  have  a  quality  assurance  manual  and  strict  control  over  all 
documentation,  data,  and  procedures  that  affect  the  quality  of  the  product. 
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The  fourth  major  AS9100  clause,  Product  Realization,  covers  the  activities  and 
processes  necessary  to  bring  a  product  into  existence.  It  is  broken  down  further  by 
the  AS9100  standard  as  follows: 

•  Planning  of  Product  Realization, 

•  Customer-Related  Processes, 

•  Design  and  Development, 

•  Purchasing, 

•  Production  and  Service  Provision,  and 

•  Control  of  Monitoring  and  Measuring  Equipment. 

Planning  of  Product  Realization  requires  the  organization  to  develop  processes 
needed  for  design  and  development  of  product  and  includes  elements  such 
as  procedures,  quality  assurance  records,  resource  requirements,  safety  and 
reliability  programs,  and  inspection  and  test.  The  Customer-Related  Processes 
section  requires  the  organization  to  determine  all  applicable  requirements 
specified  by  the  customer,  understand  those  requirements  and  associated  risks, 
and  implement  arrangements  for  communication  with  the  customer.  Design 
and  Development  includes  requirements  that  cover  planning,  inputs,  outputs, 
review,  verification,  validation,  and  control  of  changes  as  related  to  design  and 
development.  Purchasing  requires  the  organization  to  ensure  the  purchased 
product  conforms  to  specified  purchase  requirements  and  all  products  purchased 
from  suppliers  are  verified  against  these  requirements.  Production  and  Service 
Provision  requires  the  organization  to  ensure  that  production  is  accomplished 
under  controlled  conditions  using  drawings  and  specifications,  work  instructions, 
production  tools  and  software  programs,  and  provide  evidence  that  all  production 
and  inspection/verification  operations  have  been  completed  as  planned.  Control 
of  monitoring  and  measuring  equipment  requires  the  organization  to  establish, 
implement,  and  maintain  a  process  for  managing,  monitoring,  and  measuring 
equipment  that  requires  calibration  or  verification. 

The  last  major  AS9100  clause,  Measurement,  Analysis,  and  Improvement,  requires 
the  organization  to  ensure  the  product  continuously  improves.  This  clause  includes 
customer  satisfaction,  internal  audit,  monitoring  and  measuring  processes  and 
product,  and  control  of  nonconforming  products  to  ensure  continual  improvement. 
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Statutory  and  Regulatory  Requirements 

Additionally,  we  evaluated  the  F-35  Program’s  compliance  with  applicable 
statutory  and  regulatory  requirements  to  include  the  Federal  Acquisition 
Regulation;  Defense  Federal  Acquisition  Regulation  Supplement;  Interim  DoD 
Instruction  5000. 02, 7  "Operation  of  the  Defense  Acquisition  System,"  enclosure  1; 
and  DoD  4140.01,  "DoD  Supply  Chain  Materiel  Management  Procedures:  Materiel 
Sourcing,"  volume  3. 

Aviation  Critical  Safety  Items  Requirements 

Furthermore,  we  evaluated  the  F-35  Program’s  implementation  of  aviation 
CSI  requirements.  Special  attention  should  be  paid  to  CSIs  to  prevent  the  potential 
catastrophic  or  critical  consequences  of  failure.  CSIs  require  special  handling, 
engineering,  manufacturing,  and  inspection  documentation  to  control  and  ensure 
safety  of  flight. 

Public  Law  108-136,  Section  802,  "Quality  control  in  procurement  of  aviation 
CSIs  and  related  services,”  requires  the  DoD  to  prescribe  a  quality  control 
policy  for  the  procurement  of  aviation  CSIs.  The  Joint  Service  CSI  Instruction, 
"Management  of  Aviation  Critical  Safety  Items,"  implements  the  DoD  CSI  Program 
and  establishes  the  policies,  procedures,  and  responsibilities  to  manage  CSIs. 

The  Joint  Aeronautical  Commanders  Group  (JACG),  "JACG  Aviation  Critical  Safety 
Item  Handbook,"  supplements  the  Joint  Service  CSI  Instruction  and  provides 
implementation  guidance  which  describes  the  technical  and  quality  assurance 
requirements  for  a  prime  contractor  CSI  Program. 

Use  of  Computer-Processed  Data 

We  did  not  use  computer-processed  data  to  perform  this  evaluation. 

Use  of  Technical  Assistance 

We  obtained  assistance  from  quality  assurance  engineers  and  quality 
assurance  specialists  with  a  background  in  defense  and  aerospace  systems. 

We  established  teams  of  subject  matter  experts  who  evaluated  the  F-35  Program 
to  the  AS9100C,  “Quality  Management  Systems  -  Requirements  for  Aviation,  Space 
and  Defense  Organizations/'  standard.  The  subject  matter  expert  teams  consisted 
of  17  quality  assurance  engineers,  trained  and  certified  in  AS9100,  who  had  an 
average  of  17  years  of  quality  assurance,  audit  experience.  Additionally,  our  teams 
included  a  subject  matter  expert  in  military  aviation  CSI. 


7  At  the  time  of  our  evaluation,  DoDI  5000.02  was  in  interim  status.  The  final  DoDI  5000.02  was  issued  on  January  7,  2015. 
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Appendix  B 

Prior  Coverage 

The  Government  Accountability  Office  (GAO)  and  the  Department  of  Defense 
Inspector  General  (IG)  have  issued  32  reports  discussing  the  F-35  Joint  Strike  Fighter. 
Unrestricted  GAO  reports  can  be  accessed  at  http://www.gao.gov.  Unrestricted 
DoD  IG  reports  can  be  accessed  at  http://www.dodig.mil/pubs/index.cfm. 

GAO 

Report  No.  GAO-14-778,  "F-35  Sustainment:  Need  for  Affordable  Strategy,  Greater 
Attention  to  Risks,  and  Improved  Cost  Estimates,”  September  23,  2014 

Report  No.  GAO-14-340SP,  "Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,”  March  31,  2014 

Report  No.  GA0-14-468T,  "F-35  Joint  Strike  Fighter:  Slower  Than  Expected  Progress 
in  Software  Testing  May  Limit  Initial  Warfighting  Capabilities,"  March  26,  2014 

Report  No.  GAO-14-322,  "F-35  Joint  Strike  Fighter:  Problems  Completing  Software 
Testing  May  Hinder  Delivery  of  Expected  Warfighting  Capabilities,"  March  24,  2014 

Report  No.  GAO-13-690T,  "F-35  Joint  Strike  Fighter:  Restructuring  Has  Improved  the 
Program,  but  Affordability  Challenges  and  Other  Risks  Remain,"  June  19,  2013 

Report  No.  GAO-13-500T,  "F-35  Joint  Strike  Fighter:  Program  Has  Improved  in  Some 
Areas,  but  Affordability  Challenges  and  Other  Risks  Remain,”  April  17,  2013 

Report  No.  GAO-13-294SP,  "Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,”  March  28,  2013 

Report  No.  GAO-13-309,  "F-35  Joint  Strike  Fighter:  Current  Outlook  Is  Improved,  but 
Long-Term  Affordability  Is  a  Major  Concern,”  March  11,  2013 

Report  No.  GAO-12-437,  "Joint  Strike  Fighter:  DOD  Actions  Needed  to  Further 
Enhance  Restructuring  and  Address  Affordability  Risks,"  June  14,  2012 

Report  No.  GAO-12-400SP,  "Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,”  March  29,  2012 

Report  No.  GAO-12-525T,  "Joint  Strike  Fighter:  Restructuring  Added  Resources  and 
Reduced  Risk,  but  Concurrency  Is  Still  a  Major  Concern,"  March  20,  2012 
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Report  No.  GAO-11-903R,  "Joint  Strike  Fighter:  Implications  of  Program 
Restructuring  and  Other  Recent  Developments  on  Key  Aspects  of  DOD’s  Prior 
Alternate  Engine  Analyses,"  September  14,  2011 

Report  No.  GAO-11-677T,  "Joint  Strike  Fighter:  Restructuring  Places  Program  on 
Firmer  Footing,  but  Progress  Is  Still  Lagging,”  May  19,  2011 

Report  No.  GAO-11-325,  "Joint  Strike  Fighter:  Restructuring  Places  Program  on 
Firmer  Footing,  but  Progress  Still  Lags,"  April  7,  2011 

Report  No.  GAO-11-450T,  "Joint  Strike  Fighter:  Restructuring  Should  Improve 
Outcomes,  but  Progress  Is  Still  Lagging  Overall,”  March  15,  2011 

Report  No.  GAO-11-323R,  "Tactical  Aircraft:  Air  Force  Fighter  Force  Structure 
Reports  Generally  Addressed  Congressional  Mandates,  but  Reflected  Dated  Plans 
and  Guidance,  and  Limited  Analyses,"  February  24,  2011 

Report  No.  GAO-11-171R,  "Defense  Management:  DOD  Needs  to  Monitor  and  Assess 
Corrective  Actions  Resulting  from  Its  Corrosion  Study  of  the  F-35  Joint  Strike 
Fighter,”  December  16,  2010 

Report  No.  GAO-10-1020R,  "Joint  Strike  Fighter:  Assessment  of  DOD’s  Funding 
Projection  for  the  F136  Alternate  Engine,"  September  15,  2010 

Report  No.  GAO-10-789,  "Tactical  Aircraft:  DOD’s  Ability  to  Meet  Future 
Requirements  is  Uncertain,  with  Key  Analyses  Needed  to  Inform  Upcoming 
Investment  Decisions,"  July  29,  2010 

Report  No.  GAO-10-388SP,  "Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,"  March  30,  2010 

Report  No.  GAO-10-478T,  "Joint  Strike  Fighter:  Significant  Challenges  and  Decisions 
Ahead,"  March  24,  2010 

Report  No.  GAO-10-382,  "Joint  Strike  Fighter:  Additional  Costs  and  Delays  Risk  Not 
Meeting  Warfighter  Requirements  on  Time,”  March  19,  2010 

Report  No.  GAO-10-520T,  "Joint  Strike  Fighter:  Significant  Challenges  Remain  as 
DOD  Restructures  Program,”  March  11,  2010 

Report  No.  GAO-09-711T,  "Joint  Strike  Fighter:  Strong  Risk  Management  Essential  as 
Program  Enters  Most  Challenging  Phase,"  May  20,  2009 

Report  No.  GAO-09-326SP,  "Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,"  March  30,  2009 
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Report  No.  GAO-09-303,  "Joint  Strike  Fighter:  Accelerating  Procurement 
before  Completing  Development  Increases  the  Government’s  Financial  Risk," 

March  12,  2009 

Report  No.  GAO-08-782T,  "Defense  Acquisitions:  Better  Weapon  Program  Outcomes 
Require  Discipline,  Accountability,  and  Fundamental  Changes  in  the  Acquisition 
Environment,"  June  3,  2008 

Report  No.  GAO-08-467SP,  "Defense  Acquisitions:  Assessments  of  Selected  Weapon 
Programs,"  March  31,  2008 

Report  No.  GAO-08-569T,  "Joint  Strike  Fighter:  Impact  of  Recent  Decisions  on 
Program  Risks,”  March  11,  2008 

Report  No.  GAO-08-388,  "Joint  Strike  Fighter:  Recent  Decisions  by  DOD  Add  to 
Program  Risks,”  March  11,  2008 

DoD  IG 

Report  No.  DODIG-2013-140,  "Quality  Assurance  Assessment  of  the  F-35  Lightning  II 
Program,”  September  30,  2013 

Report  No.  DODIG-2013-031,  "Audit  of  the  F-35  Lightning  II  Autonomic  Logistics 
Information  System  (ALIS),"  December  10,  2012 
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Appendix  C 

Management  Comments  on  the  Findings 
and  Our  Responses 

Finding  A  -  Insufficient  Progress  Had  Been  Made  Toward 
Implementation  of  the  Critical  Safety  Program 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  disagreed  and  stated: 

The  F-35  JPO  places  the  safety  of  aircrew  and  maintainers  foremost  within 
all  program  priorities.  Working  with  DCMA  and  Lockheed  Martin  to  bring 
the  management  of  CSI  into  compliance  with  requirements,  progress  is  being 
made  toward  the  dates  provided  to  the  DODIG.  This  is  not  however,  as 
implied,  a  simple  matter  of  'flowing  down'  this  highly  complex  requirement. 
This  contractual  change  constitutes  an  increase  in  scope,  incorporating 
non-recurring  and  recurring  effort  into  multiple  F-35  contracts,  and 
must  be  done  in  accordance  with  contract  law,  federal  regulation  and 
fiscal  responsibility. 

Military  aviation  programs  that  existed  before  the  CSI  requirement  was 
created  have  faced  similar  challenges  implementing  the  CSI  law  based  on  wide 
interpretation  of  its  intricate  and  far  reaching  mandate.  The  F-35  Lightning  II 
System  Design  and  Development  Contract  was  signed  October  26,  2001,  by 
2003  the  basic  design  was  nearly  complete  and  fabrication  beginning,  with 
first  rollout  in  February  2006.  Even  recognizing  aircraft  design  changes  for 
weight  reduction  in  2004,  when  the  CSI  public  law  was  issued  it  constituted 
a  new  requirement  to  the  existing  contract.  Also,  implementing  such  a 
requirement  without  department  instruction  or  guidance  could  have  incurred 
significant  risk  in  implementing  it  incorrectly.  The  first  issuance  of  the 
Joint  Aeronautical  Logistics  Commanders  (JALC)  Aviation  CSI  handbook 
was  August  12,  2005  and  the  Joint  Service  Instruction  was  not  issued  until 
January  25,  2006. 

As  was  objectively  demonstrated  in  explicit  detail  to  the  DODIG  the  F-35 
JPO  and  Lockheed  Martin  jointly  developed  F-35  integrity  programs  based 
on  several  decades  of  military  aircraft  design  and  development  for  the 
United  States  armed  forces.  The  integrity  policies  and  procedures  form  one  of 
the  cornerstones  for  rigorous  qualification  of  designs  to  ensure  safety  and 
functionality  that  progressively  moves  designs  from  engineering  analysis 
(including  flight  safety  critical  aircraft  parts)  through  production  fabrication, 
testing,  and  lifecycle  management  before  incorporation  into  the  aircraft. 
Additionally,  designs  are  secondarily  analyzed  by  government  subject  matter 
experts  for  airworthiness  and  certified  to  standards  developed  specifically 
for  this  purpose.  Next,  the  qualified  parts  are  assembled  to  constitute  an 
aircraft  which  moves  into  a  series  of  reviews  and  certifications  before  first 
flight  of  a  severely  restricted  flight  envelope  that  is  progressively  opened 
through  carefully  monitored  flight  testing  conducted  by  the  contractor 
and  government. 
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The  CSI  requirement  establishes  policy,  procedure  and  assigns  responsibility 
for  the  lifecycle  management  of  replenishment  items  (spare  parts,  support 
equipment  and  repairs)  critical  to  safety.  The  primary  methodology  is  to 
assure  spare  parts  and  services  are  procured  from  approved  sources  and 
manufactured  to  correct  standards  and  processes.  The  public  law,  department 
and  service  instructions  do  not  specify  that  CSI  is  to  be  incorporated 
during  the  development  phase  of  aircraft  systems,  though  it  has  been 
recognized  within  DoD  that  the  intrinsically  necessary  failure  mode  analysis, 
documentation  of  characteristics,  development  of  quality  assurance  above  and 
beyond  standard  practice  with  primary  difference  being  removal  of  likelihood 
as  analysis  criteria  should  be  done  in  parallel  with  other  safety,  airworthiness 
and  integrity  scrutiny.  To  incorporate  CSI  in  this  manner  it  would  have  to 
be  done  during  a  contract's  initial  proposal  which  in  the  case  of  F-35  was 
several  years  before  the  CSI  law's  inception.  Therefore  the  F-35  JPO  has 
used  department  instruction  to  interpret  the  program  and  existing  drawings 
as  legacy.  The  F-35  JPO  engineering  support  activity  considers  there  to 
be  sufficient  protections  in  place  to  assure  safety  through  existing  quality 
assurance,  integrity,  qualification,  airworthiness,  and  flight  test  programs 
until  full  CSI  compliance  is  reached.  Currently  all  F-35  Lightning  II  spares 
and  repairs  are  procured  from  their  original  equipment  manufacturers  with 
inherent  quality,  integrity,  airworthiness  and  safety  assured. 


Our  Response 

JPO  is  planning  to  implement  the  CSI  Program  by  February  2016;  however,  the 
draft  CSI  Program  prepared  by  Lockheed  Martin  that  we  evaluated  did  not 
sufficiently  address  requirements  described  in  the  Joint  Service  CSI  Instruction. 
The  F-35  Program  has  not  made  sufficient  progress  toward  full  compliance 
with  CSI  Public  Law  since  it  was  issued  in  2003.  NAVAIR  Instruction  4200. 25C, 
September  1999,  had  been  established  and  in  use  prior  to  the  SDD  contract 
award  in  2001.  NAVIAR  Instruction  4200. 25D,  which  was  issued  in  June  2002, 
contains  very  similar  CSI  requirements  as  SECNAVINST  4140.2,  which  was 
released  in  January  2006.  There  are  many  other  DoD  instructions  and  guidelines 
such  as  DoD  Regulation  4140. 1-R,  May  2003  and  the  prior  version,  which  was 
issued  in  January  1993;  JALC  CSI,  August  2005;  the  Joint  Service  Instruction, 
January  2006;  and  JACG  Aviation  CSI  Management  Handbook,  March  2011; 
which  were  available  to  be  used  for  the  F-35  CSI  Program.  These  requirements 
could  have  been  implemented  in  either  the  SDD  or  LRIP  contracts,  starting  with 
LRIP  lot  1.  JPO  was  aware  of  the  existence  of  NAVAIRINST  4200. 25D  as  it  was  put 
in  the  LRIP  lot  1  contract  for  F-35  engine  development  awarded  in  April  2006. 
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JPO  believes  that  the  F-35  integrity  programs,  which  are  jointly  developed  by 
Lockheed  Martin  and  JPO,  address  all  the  CSI  requirements;  however,  JPO  was  not 
able  to  provide  any  evidence  to  show  that  it  had  performed  a  gap  analysis  between 
the  F-35  integrity  programs  and  CSI  requirements.  We  identified  gaps  during  our 
last  evaluation,  which  showed  that  the  F-35  integrity  program  did  not  cover  all 
CSI  requirements. 

JPO  stated, " [t] he  public  law,  department  and  service  instructions  do  not 
specify  that  CSI  is  to  be  incorporated  during  the  development  phase  of  aircraft 
systems...”  Contrary  to  JPO’s  claim,  JACG  Aviation  CSI  Management  Handbook, 
section  2.3.2  states: 

[tjhe  basis  of  CSI  determinations  on  repairable  and  consumable 
parts  and  maintenance  concepts  underscores  the  importance  of 
effective  working  relationships  among  logistics  and  engineering 
communities.  Logisticians  are  responsible  for  life  cycle  support 
concepts  that  ultimately  define  spare  parts  and  for  providing  these 
concepts  to  engineers  who  are  then  responsible  for  identifying 
CSIs  and  approving  their  sources.  This  relationship  is  particularly 
meaningful  during  the  acquisition,  development,  and  initial 
production  stages  of  new  platforms. 

Section  2.9.3  of  JACG  Aviation  CSI  Management  Handbook  further  states, 
"[tjherefore,  a  requirement  for  prime  contractor/OEM's  to  identify  critical 
characteristics  for  every  new  CSI  should  be  included  in  the  System 
Development  and  Demonstration  Statement  of  Work  for  new  systems 
and  major  platform  modifications.” 

JPO’s  decision  to  postpone  the  implementation  of  CSI  requirements  may  increase 
cost,  schedule,  and  safety  risks.  Significantly  more  efforts  and  resources  will 
be  required  to  accommodate  CSI  requirements  if  they  are  implemented  in  the 
later  stages  of  the  acquisition  lifecycle:  such  as  performing  engineering  analysis, 
identifying  critical  characteristics,  updating  drawings,  and  creating  a  logistics 
process  to  support  the  changes.  JPO  did  not  explain  how  it  is  more  fiscally 
responsible  to  delay  incorporating  contractual  CSI  requirements.  In  addition, 

JPO  had  many  opportunities  to  implement  CSI  requirements  during  program 
restructuring,  which  took  place  in  December  2003,  again  in  March  2007,  and  most 
recently  in  March  2012. 
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Finding  B  -  Oversight  of  Product  Development  and  Realization 
of  Requirements  was  Inadequate 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  disagreed  and  stated: 

The  F-35  JPO  has  progressively  made  improvements  to  aircraft  acceptance 
criteria.  In  LRIP  6,  as  in  previously  LRIPs,  known  JCS  non-compliances 
were  documented.  It  is  true  that  at  aircraft  acceptance  F-35  JPO  does 
not  require  LM  to  positively  verify  compliance  to  every  JCS  requirement. 

This  is  due  to  the  concurrent  nature  of  SDD  and  Production  schedules. 

The  SDD  schedules  identify  when  each  JCS  requirement  is  expected  to  meet 
compliance  criteria.  The  vast  majority  of  compliance  occurs  in  the  2016  and 
2017  timeframe.  If  F-35  JPO  were  to  impose  JCS  compliance  assessments  in 
production,  prior  to  expected  SDD  JCS  compliance  dates,  the  assessments 
would  be  based  on  subjective  rather  than  objective  criteria  because  proper 
SDD  certification  evidence  would  not  be  available.  The  F-35  JPO  deemed  that 
subjective,  premature  JCS  compliance  assessments  were  not  value  add  for  the 
program.  Additionally,  these  subjective  assessments  would  negatively  impact 
SDD  resources  (people,  labs  and  flight  test  assets)  and  schedules. 

Our  Response 

Acquisition  strategy  (concurrent  development)  should  not  prevent  JPO  from 
establishing  objective  and  measurable  aircraft  acceptance  criteria.  The  lack  of 
objective  and  measurable  aircraft  acceptance  criteria  in  the  CCDD  was  documented 
during  our  last  evaluation.  This  nonconformity  was  agreed  to  by  Lockheed  Martin 
and  JPO;  however,  JPO  still  has  not  addressed  it  adequately.  This  same  deficiency 
was  documented  again  during  this  evaluation.  Our  report  states  “...JPO  and 
Lockheed  Martin  are  at  an  impasse  to  perform  implementation  of  corrective 
action  due  to  a  breakdown  in  contract  negotiations."  As  a  result,  there  has  not 
been  sufficient  progress  in  defining  objective  and  measureable  aircraft  acceptance 
criteria.  Our  finding  does  not  state  that  the  F-35  LRIP  aircraft  need  to  be 
tested  at  full  JCS  compliance.  Rather,  the  CCDD  acceptance  criteria  for  a  specific 
LRIP  lot  should  be  measurable  and  testable.  We  understand  that  production 
and  development  efforts  are  concurrent;  however,  it  does  not  relieve  JPO  of  the 
responsibility  for  identifying  clear  and  measureable  acceptance  criteria  for  the 
LRIP  aircraft. 
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Finding  C  -  The  Quality  Assurance  Organization  Was  Not 
Independent  and  Not  Adequately  Staffed 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  partially  agreed  and  stated: 

The  F-35  JPO  does  not  place  schedules  ahead  of  quality.  Further,  the  F-35  JPO 
has  withheld  delivery  of  aircraft  to  assure  quality  issues  are  addressed. 
Although  the  F-35  JPO  deems  the  current  reporting  structure  adequate  to  meet 
program  needs,  it  has  agreed  to  create  a  more  robust  quality  organization  to 
ensure  quality  management  is  objectively  measured  and  reported. 


Our  Response 

The  F-35  Program  quality  assurance  organization  still  resides  under  the  Director 
of  Production  and  is  not  a  direct  report  to  the  F-35  PEO.  We  recommend  that 
JPO  work  toward  the  organizational  structure  defined  in  JPO’s  response  letter 
to  the  DODIG-2013-140  report  dated  August  8,  2014.  The  response  letter  stated, 
"[t]he  F-35  JPO  will  establish  an  independent  quality  organization  reporting  to  the 
Program  Executive  Officer."  However,  during  the  2014  evaluation,  JPO’s  planned 
quality  organization  reported  to  the  Director  of  Production,  which  we  do  not 
consider  to  be  independent.  JPO  should  realign  its  quality  organization  in 
accordance  with  its  August  8,  2014,  response. 

Finding  D  -  Reduction  of  the  Assembly  Defect  Rate 
Was  Inadequate 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  disagreed  and  stated: 

The  program  quality  90%  objective  the  DODIG  references  was  a  ratio  of  Scrap 
Rework  and  Repair  (SRR)  to  production  labor  hours  (hence  a  'rate').  This 
is  a  normalization  technique  to  assess  levels  of  SRR  relative  to  volume  of 
work  in  a  given  area.  This  normalization  is  a  valuable  metric;  however,  it  is 
not  the  only  metric.  In  the  current  case  of  F-35  the  production  line  which  is 
still  experiencing  significant  production  labor  hour  reductions  as  a  result  of 
progressing  down  predictable  learning,  the  ratio  of  declining  SRR  hours  to 
declining  production  labor  hours  has  remained  relatively  steady.  The  F-35  JPO, 
DCMA  and  LM  have  worked  to  understand  this  phenomenon  and  analyzed 
other  metrics  to  evaluate  the  program's  progress  toward  reductions  necessary 
to  reach  high  production  rates  by  increasing  efficiency  and  flow. 

•  Factory  SRR  hours  have  decreased  65%  since  the  first  production  lot. 
Supplier  responsible  SRRs  have  been  reduced  80%  and  the  program  is 
setting  a  goal  to  reduce  the  current  level  by  another  25%  again  this  year. 

•  Quality  Escapes  to  the  field  are  another  measure,  which  have  decreased 
75%  since  2011. 
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Our  Response 

We  appreciate  the  additional  clarification  on  the  full  rate  production  objective 
identified  in  the  F-35  Program  Quality  Management  Plan.  However,  the  Quality 
Management  Plan  does  not  provide  any  reference  to  the  use  of  SRR  to  determine 
the  assembly  defect  rate.  Rather,  it  depicted  a  reduction  in  the  number  of  quality 
assurance  reports  per  aircraft  per  LRIP  lot  as  a  reduction  in  the  defect  rate,  which 
is  not  equivalent. 

Finding  E  -  Corrective  Action  Request  Escalation 
Was  Inadequate 

DCMA  Comments 

DCMA  agreed  and  stated: 

DCMA  agrees  we  did  not  execute  to  DCMA  Instruction  #1201  as  interpreted 
by  the  DoD  IG  team.  We  agree  that  DCMA  instruction  1201  requires 
clarification.  It  is  currently  under  revision  to  address  clarification  to  the 
CAR  elevation  process. 

DCMA  believes  this  risk  is  mitigated  based  on  a  comprehensive  program 
quality  strategy  between  DCMA,  F-35  JPO,  and  LM.  There  is  a  monthly 
metrics  cadence  in  which  the  DCMA  Quality  organization  provides 
independent  CAR  analysis.  The  data  presented  in  these  meetings  provides 
insight  of  non-compliances  identified  through  DCMA  and  LM  Quality 
surveillance  activities. 


Our  Response 

Comments  from  DCMA  meet  the  intent  of  the  finding. 

Finding  F  -  Software  Quality  Management  Was  Insufficient 

F-35  Joint  Program  Office  Comments 

F-35  Joint  Program  Office  disagreed  and  stated: 

The  F-35  JPO  considers  F-35  software  quality  management  is  sufficient, 
specifically  the  air  vehicle  software  quality  referenced  in  the  draft  DOD  IG 
audit  report.  The  F-35  JPO  is  confident  current  air  vehicle  software 
configuration  management,  failure  reporting  and  corrective  action,  root  cause 
analysis,  and  training  are  sufficient  and  adequate  to  support  program  mission 
capability  and  cost  objectives.  The  F-35  JPO  is  confident  in  the  current 
air  vehicle  software,  systems  engineering,  and  airworthiness  certification 
processes  will  ensure  no  undiscovered  software  defects  will  be  delivered 
with  the  air  vehicle  that  could  result  in  performance  degradation  or  even 
catastrophic  failure  of  aircraft. 
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Specific  DoD  IG  Finding  Responses: 

1.  DoD  IG  Finding:  Lockheed  Martin's  software  development  methodology 
did  not  facilitate  the  performance  of  physical  and  functional 
configuration  audits.  The  majority  of  new  aircraft  capability  is  based 
on  additional  software  builds;  therefore,  it  is  necessary  to  perform 
functional  and  physical  configuration  audits.  These  types  of  audits 
ensure  that  CSCIs  exist  and  are  configured,  tested,  and  documented  in 
accordance  with  software  specifications  and  requirements  to  mitigate 
aircraft  performance  and  reliability  risks. 

F-35  JPO  Response:  In  accordance  with  the  F-35  program  Software 
Development  Plan  and  Configuration  Management  Plan,  physical 
and  functional  configuration  audits  of  the  type  referenced  by  the 
DOD  IG  report  are  conducted  against  the  air  vehicle  Software 
Data  Load,  which  is  a  compilation  of  vehicle  and  mission  systems 
components.  The  program  is  conducting  FCA/PCAs  in  accordance 
with  the  F-35  CM  Plan,  2YZA00002,  on  all  CIs  and  CSCIs  identified  in 
the  F-35  Cl  List,  2YZA00941,  approved  by  the  F-35  JPO.  This  finding 
erroneously  presumes  that  certain  software  items  which  are  not 
identified  in  the  Cl  list  are  CSCIs  and  therefore  must  be  subject  to 
FCA/PCA  requirements.  Mission  System  and  Vehicle  System  computer 
programs  are  elements  which  contribute  to  the  composition  of  the 
aircraft  Software  Data  Load.  It  is  the  aircraft  Software  Data  Load 
which  is  identified  in  the  F-35  Cl  List  as  a  CSCI,  and  it  is  this  software 
product  on  which  FCA/PCA  activity  will  be  conducted.  This  practice 
is  consistent  with  software  development  practices  on  all  weapon 
systems  developed  at  the  Fort  Worth  facility,  including  the  F-16  and 
F-22,  in  accordance  with  the  DoD  required  Single  Process  Initiative 
for  the  Fort  Worth  Facility.  The  F-35  program  has  a  very  rigorous  and 
structured  verification  and  validation  process  for  software  enabled 
systems  capabilities,  including  the  identification  and  management  of 
all  software  components  that  contribute  to  safety  critical  functions. 

In  addition,  F-35  air  vehicle  software  undergoes  extensive  independent 
reviews  by  external  airworthiness  agencies  from  the  United  States 
Air  Force  and  Navy  in  the  context  of  MIL-HDBK-516,  as  well  as  the 
Ministries  of  Defense  of  the  United  Kingdom  and  Australia.  These 
agencies  have  independently  confirmed  the  adequacy  of  the  physical 
configuration  and  functional  ability  of  the  air  vehicle  software  for  safe 
operation  in  the  delivered  operational  envelope.  The  F-35  program 
does  not  intend  to  modify  existing  functional  and  physical  configuration 
audit  practices  for  F-35  air  vehicle  software  based  on  this  finding. 
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2.  DoD  IG  Finding:  Deficiencies  in  Lockheed  Martin's  software 
development  training  program  may  lead  to  knowledge  gaps  which  could 
result  in  software  development  process  failures  and  software  defects. 

F-35  JPO  Response:  Lockheed  Martin  has  a  structured  and  accountable 
training  program  for  all  software  engineers  in  the  LM  aeronautical 
division,  documented  in  PM-4002  (Software  Engineering  Training). 

In  addition,  the  LM  F-35  program  has  a  structured  and  accountable 
training  program  for  F-35  specific  software  development  environments 
and  tools.  Training  history  for  both  LM  Aero  and  F-35  specific  training 
were  provided  to  the  IG  team,  as  well  as  learning  plans  for  specific 
classes  of  software  engineering  disciplines.  Air  vehicle  software 
engineers  on  the  F-35  program  comprise  a  body  of  world  class 
talent  -  they  are  industry  experts  in  the  development  of  hard  real 
time  embedded  flight  and  mission  critical  software,  and  have  in  most 
cases  decades  of  experience  in  vehicle  and  mission  systems  software 
development  on  other  4th  and  5th  generation  fighter  programs.  They 
are  sought  after  by  industry  to  provide  instruction  to  other  engineers 
in  their  discipline  areas,  and  work  directly  with  the  developers  of 
the  software  tools  and  environments  they  are  using  to  incorporate 
additional  enhancements  and  features  to  increase  productivity  and 
efficiency.  The  F-35  JPO  does  not  intend  to  pursue  any  additional 
training  initiatives  for  the  LM  software  development  team  based  on 
this  finding. 

3.  DoD  IG  Finding:  The  classification  of  the  severity  of  software 
deficiencies  identifies  the  level  of  verification  and  corrective  action 
required.  Lockheed  Martin's  failure  to  ensure  software  subcontractors, 
such  as  Northrop  Grumman,  were  categorizing  software  deficiencies 

in  accordance  with  the  F-35  Air  System  Software  Development  Plan 
does  not  provide  confidence  in  delivered  software.  Allowing  open 
deficiencies,  especially  those  whose  severities  are  not  characterized 
and  understood,  can  result  in  performance  degradation  or  catastrophic 
failure  during  operation. 

F-35  JPO  Response:  System  and  software  deficiency  severity 
classification  and  tracking  is  managed  and  maintained  at  the  air  vehicle 
system  level,  in  accordance  with  F-35  program  systems  engineering, 
software  engineering,  and  system  verification  plans.  The  DoD  IG 
team  assumed  subcontractors  would  independently  classify  and  track 
their  anomalies  using  the  same  criteria  as  the  air  vehicle  teams,  which 
is  not  the  defined  process  or  practice.  Subcontractor  deficiencies 
are  reported  against  the  specification  requirements  levied  to  the 
subcontractors,  but  cannot  be  categorized  by  the  subcontractor  team  in 
terms  of  air  vehicle  severity,  as  only  the  air  vehicle  systems  engineering 
teams  in  the  vehicle  systems  and  mission  systems  functional  areas 
have  the  required  breadth  and  scope  of  system  functionality  and  safety 
criticality  to  adequately  classify  the  severity  of  component  anomalies 
in  the  context  of  air  vehicle  operations.  The  F-35  JPO  reviewed  the 
specific  concerns  highlighted  by  the  DoD  IG  with  Northrop  Grumman 
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failure  reporting,  and  found  the  reporting  practice  consistent  with  the 
documented  air  vehicle  software,  systems  engineering,  and  air  vehicle 
verification  plan.  There  are  no  open  deficiencies  delivered  with  the  air 
vehicle  that  are  not  characterized  by  hazard  severity  and  understood 
within  the  context  of  mission  and  safety  criticality,  and  which  have 
not  been  adjudicated  with  the  F-35  JPO,  airworthiness  authorities,  and 
operational  users.  The  F-35  JPO  does  not  plan  to  modify  the  existing 
software,  systems  engineering,  and  system  verification  process  or 
practices  based  on  this  finding. 


Our  Response 

We  do  not  agree  that  the  F-35  Program’s  software  quality  management  program 
sufficiently  meets  the  requirements  of  AS9100  and  that  no  further  corrective  action 
is  required. 

At  the  Air  Vehicle  Software  Data  Load  (SDL)  level,  the  FCA/PCA  does  not  include  all 
prime  mission  software  configuration  items  such  as:  Fusion  Software,  Pilot  System 
Software,  Fire  Controls/Navigation/Stores  (MSFCS),  Mission  Domain  (MSMIS), 

Data  Collection  Software,  External  Communication  Software  (MSXCM),  and  Core 
Processing  Software.  Eliminating  the  FCA/PCA  program  at  the  software  level 
prevents  the  critical  verification  process  and  significantly  impacts  software 
product  integrity.  The  intent  of  FCA/PCA  cannot  possibly  be  met  at  the  Air  Vehicle 
SDL  level. 

Although  Lockheed  Martin’s  training  program  did  include  position  specific 
training  requirements,  we  did  not  find  any  evidence  that  Lockheed  Martin’s 
software  integrated  product  teams  identified  and  included  project-specific  training 
requirements  within  software  engineer  training  curriculums.  We  also  did  not  find 
any  evidence  that  Lockheed  Martin  Integrated  Product  Team  leads  had  reviewed 
training  curricula  annually  as  required  by  Lockheed  Martin’s  command  media. 

Finally,  we  agree  with  JPO’s  statement  that  F-35  Air  System  software  deficiency 
severity  levels  may  not  apply  directly  to  software  defects  within  lower  level 
software  components.  However,  the  Air  System  SDP  stated  "[pjroduct-level  SDPs 
must  be  created  or  updated  to  conform  to  processes  identified  in  this  version  of 
the  AS  [Air  System]  SDP."  This  identifies  that  the  subcontractors  SDP  must  comply 
with  Air  System  SDP  processes,  which  would  include  the  categorization  of  software 
deficiencies  to  alleviate  translation  conflicts  of  the  F-35  Air  System  software 
deficiency  severity  levels. 
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Management  Comments 

F-35  Joint  Program  Office  and  Defense  Contract 
Management  Agency  Comments 


F-35  LIGHTNING  II  JOINT  PROGRAM  OFFICE 
200  I2,b  Street  South,  Suite  600 
Arlington,  Virginia  22202-5402 

JAN  20  2015 

From:  Program  Executive  Officer,  F-35  Lightning  [J  Joint  Program  Office  (JPO)  and  Director, 
Defense  Contract  Management  Agency  (DCMA) 

To;  Department  of  Defense  (DoD)  Deputy  Inspector  General  (1G)  tor  Policy  and  Oversight 

Subj:  RESPONSE  TO  F-35  LIGHTNING  II  PROGRAM  QUALITY  ASSURANCE  AND 
CORRECTIVE  ACTION  EVALUATION  (Project  No.  D2G14-OTOTAD-OOQ4) 

Thank  you  for  the  opportunity  to  comment  on  your  draft  report,  ‘‘Quality  Assurance 
Assessment  of  the  F-35  Lightning  II  Program"*  dated  23  December  2014. 

Both  the  F-35  Lightning  IF  Joint  Strike  Fighter  Program  Office  (JPO)  and  the  Defense 
Contract  Management  Agency  (DCMA)  take  very  seriously  the  findings  and  recommendations 
offered  in  the  report,  As  noted  in  Enclosure  l ,  F-35  JPO  and  DCMA  Response  to  Findings  and 
Recommendations,  our  learn  has  aggressively  addressed  DoD  1G  findings  since  receipt  of  the 
first  report  in  September  2013,  The  implementation  of  these  corrective  actions  has  led  to 
significant  progress  in  delivering  a  quality,  timely,  and  cost-effective  weapon  system  to  the 
United  Stales  and  its  allies. 


Enclosure  l  provides  a  detailed  response  to  each  finding  and  recommendation. 

Enclosure  2  is  a  copy  of  a  Lockheed  Martin  letter  provided  to  the  F-35  JPO  detailing  the  quality 
improvement  made  since  issuance  of  the  September  201 3  report. 

_ The  t.  if II- ITS  Mi  1  |  ■  i  1 H  S  ■.  -ir<J  £'&?>*( 'J 


Lieutenant  General,  USAF 
Program  Executive  Officer 


WENDY  M.  MASIELLO 
Lieutenant  General,  USAF 
Defense  Contract  Management  Agency 


Enclosures: 

3 ,  F-35  JPG  and  DCMA  Response  to  Findings/Rccommcndations 
2.  LM  Letter  to  F-35  JPO  on  Quality  Improvements 


uc: 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Assistant  Secretary  ofthe  Navy  for  Research,  Development  and  Acquisition 
Assistant  Secretary  ofthe  Air  Force  (Acquisition) 
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F-35  Joint  Program  Office  and  Defense 
Contract  Management  Agency  Response  to 
Findings/Recommendations 
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Finding  A 

Insufficient  Progress  Had  Been  Made  Toward 
Implementation  of  the  Critical  Safety  Item  Program 

The  F-35  Program  had  not  made  sufficient,  progress  towards  full  compliance  with  Public  haw  i  OB¬ 
IS  6,  Section  802,  "Quality  control  in  procurement  of  aviation  CSls  and  related  services,"  and  the 
Joint  5ervice  CSI  Instruction,  "Management  of  Aviation  Critical  Safety  Items/'  by  February  2  016. 

This  was  caused  by  the  lack  of  CSI  requirements  on  the  F-35  contracts,  as  required  by  public  law 
and  the  |omL  Service  CSI  Instruction.  This  lack  of  compliance  with  CSI  public  law  and  implementing 
instructions  may  impose  an  increased  safety  of  flight  risk  on  delivered  aircraft 

Discussion 

As  identified  in  our  previous  report  ]PO  had  not  flowed  down  a  contractual  requirement  to 
Lockheed  Martin  to  implement  a  CSI  Program  in  accordance  with  Public  Law  108-  136,  Section  802 
and  the  joint  Service  CSI  Instruction.  The  CSI  public  law  has  been  imposed  since  2003. 

JPOand  Lockheed  MarLin  have  taken  iniLiat  steps  to  establish  a  CSI  Program;  however  Lhe  program 
was  sLifl  not  fully  compliant  with  applicable  laws  and  regulations.  JPO  was  planning  to  contractually 
require  im  piemen  Lallan  of  a  CSI  Program  by  February  2016. 

However,  the  current  draft  of  the  CSI  Program  prepared  hy  Lockheed  Martin  did  not  sufficiently 
address  requirements  described  In  the  Joint  Service  CSI  Instruction  including  the  need  to  define 
critical  characteristics  for  all  CSls  and  the  proper  limitations  and  delegations  for  the  material 
review  board  authority  far  nonconforming  CSI  critical  characteristics. 

F-35  JPO  Response:  Disagree 

The  F-35  JPO  places  the  safety  of  aircrew  and  maintainers  foremost  within  all  program 
priorities.  Working  with  DCMA  and  Lockheed  Marlin  to  bring  the  management  of  CSI 
into  compliance  with  requirements,  progress  is  being  made  toward  the  dates  provided  to 
the  DODIG.  This  is  not  however,  as  implied,  a  simple  matter  of  ‘flowing  down1  this 
highly  complex  requirement  This  contractual  change  constitutes  an  increase  in  scope, 
incorporating  non-recurring  and  recurring  effort  into  multi  pie  F-35  contracts,  and  must 
be  done  in  accordance  with  contract  law,  federal  regulation  and  fiscal  responsibility. 

Military  aviation  programs  that  existed  before  the  CSI  requirement  was  created  have 
faced  similar  challenges  implementing  the  CSI  law  based  on  wide  interpretation  of  its 
intricate  and  far  reaching  mandate.  The  F-35  Lightning  II  System  Design  and 
Development  Contract  was  signed  October  26,  2001,  by  2003  the  basic  design  was 
nearly  complete  and  fabrication  beginning,  with  first  rollout  in  February  2006.  Even 
recognizing  aircraft  design  changes  for  weight  reduction  in  2004,  when  the  CSI  public 
law  was  issued  it  constituted  a  new  requirement  to  the  existing  contract.  Also, 
implementing  such  a  requirement  without  department  instruction  or  guidance  could 
have  incurred  significant  risk  in  implementing  it  incorrectly.  The  first  issuance  of  the 
Joint  Aeronautical  Logistics  Commanders  (JALC)  Aviation  CSI  handbook  was  August 
12,  2005  and  the  Joint  Service  Instruction  was  not  issued  until  January  25,  2006 
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F-35  Joint  Program  Office  and  Defense 
Contract  Management  Agency  Response  to 
Findings/Recommendations  (cont'd) 


‘T  I  i  i  m  nij  w  . 1 1 ' h i r  t  rcUVft  Aci  1th  , "  i 

Pn#ji Wt>;  nZ0l4‘nTflTAD*fl(IO44Kia 
F-U5  )!"0  inti  DLMA  Ki^p*in9C& 

As  was  objectively  demonstrated  in  explicit  detail  to  the  DODIG  the  F-35  JPO  and 
Lockheed  Martin  jointly  developed  F-35  integrity  programs  based  on  several  decades  of 
military  aircraft  design  and  development  for  the  United  States  armed  forces  The 
integrity  policies  and  procedures  form  one  of  the  cornerstones  for  rigorous  qualification 
of  designs  to  ensure  safety  and  functionality  that  progressively  moves  designs  from 
engineering  analysis  (including  flight  safety  critical  aircraft  parts)  through  production 
fabrication,  testing,  and  lifecycle  management  before  incorporation  into  the  aircraft 
Additionally,  designs  are  secondarily  analyzed  by  government  subject  matter  experts  for 
airworthiness  and  certified  to  standards  developed  specifically  for  this  purpose.  Next, 
the  qualified  parts  are  assembled  to  constitute  an  aircraft  which  moves  into  a  series  of 
reviews  and  certifications  before  first  flight  of  a  severely  restricted  flight  envelope  that  is 
progressively  opened  through  carefully  monitored  flight  testing  conducted  by  the 
contractor  and  government, 

The  CSI  requirement  establishes  policy,  procedure  and  assigns  responsibility  for  the 
lifecycle  management  of  replenishment  items  (spare  parts,  support  equipment  and 
repairs)  critical  to  safety.  The  primary  methodology  is  to  assure  spare  parts  and 
services  are  procured  from  approved  sources  and  manufactured  to  correct  standards 
and  processes  The  public  law,  department  and  service  instructions  do  not  specify  that 
CSI  is  to  be  incorporated  during  the  development  phase  of  aircraft  systems,  though  it 
has  been  recognized  within  DoD  that  the  intrinsically  necessary  failure  mode  analysis, 
documentation  of  characteristics,  development  of  quality  assurance  above  and  beyond 
standard  practice  with  primary  difference  being  removal  of  likelihood  as  analysis  criteria 
should  be  done  in  parallel  with  other  safety,  airworthiness  and  integrity  scrutiny  To 
incorporate  CSI  in  this  manner  it  would  have  to  be  done  during  a  contract's  initial 
proposal  which  in  the  case  of  F-35  was  several  years  before  the  CSI  law's  inception 
Therefore  the  F-35  JPO  has  used  department  instruction  to  interpret  the  program  and 
existing  drawings  as  legacy.  The  F-35  JPO  engineering  support  activity  considers  there 
to  be  sufficient  protections  in  place  to  assure  safety  through  existing  quality  assurance, 
integrity,  qualification,  airworthiness,  and  flight  test  programs  until  full  CSt  compliance  is 
reached  Currently  all  F-35  Lightning  tl  spares  and  repairs  are  procured  from  their 
original  equipment  manufacturers  with  Inherent  quality,  integrity,  airworthiness  and 
safety  assured. 


Recommendations 

A  We  recommend  that  the  F-35  |oinL  Program  Office: 

L  Ensure  that  the  F-35  CSJ  Program  is  compliant  with  Public  Law  108  136,  Section  802,  "Quality 
control  in  procurement  of  aviation  CSls  and  related  services/'  and  the  joint  Service  CSI  Instruction, 
''Management  of  Aviation  Critical  Safety  Items/1 

F-35  JPO  Response!  Agree 
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F-35  Joint  Program  Office  and  Defense 
Contract  Management  Agency  Response  to 
Findings/Recommendations  (cont'd) 


F-35  Lightning  II  Program  Quality  Assurance  and  Corrective  Action  Evaluation 
Project  No.  D2G14-DTOTAD-OOO4.O0O 
F-35  JPO  and  DCMA  Findings,  Recommendations  and  Responses 

-  Contract  actions  for  both  SDD  and  LRIP  that  address  the  Joint  Service  CSI 
Instruction,  “Management  of  Aviation  CSIs”  are  in  work  or  in  place  to  implement  the 
full  F-35  CSI  Program 

o  SDD  RFP  for  CSI  NRE  efforts  to  initiate  F-35  CSI  Program  released  Oct  2013  and 
resulted  in  a  LM  delivered  Draft  CSI  Program  Plan  coordinated  with  F-35  JPO  for 
approval  of  the  CSI  implementation  approach  and  go  ahead  for  delivery  of  the  SDD 
proposal  expected;  go  ahead  date  for  proposal  development  expected  2QFY15 

-  After  the  SDD  CSI  NRE  efforts  are  completed,  the  full  scope  of  the  F-35  CSI 
program  will  be  included  in  future  production  and  sustainment  contracts. 

o  LRIP  Production  contracts  (LRIP  1-8)  currently  cover  explicit  CSI  tasks  for  "limited" 
compliance  for  several  CSI  ‘gaps’  from  the  Joint  Service  CSI  Instruction, 
Management  of  Aviation  CSIs." 

-  Maintain  /  update  the  DOE  approved  CSI  list  for  configuration  control 

-  Use  existing  recurring  processes:  Material  Review  (MR),  Variance,  item 
Acceptance,  New  Suppliers,  Process  Metrics,  and  Minor  Si  Major  Change 
processes  for  CSIs 

-  Provide  CSI  notification  to  affected  suppliers  via  contracts  letter 

-  Provide  notification  if  any  identified  non-conformances  results  in  a  safety  impact 

-  Report  MR  actions  &  QARs  associated  with  CSI  once  per  month 

-  Use  existing  disposal  processes  to  destroy  scrapped  CSIs  that  render  part 
nonfunctional 

-  DCMA-LM  FW  Integrated  CSI  Surveillance  Strategy  developed  Jan  2014  and  covers 
CSI  Delegations  to  ensure  appropriate  DCMA  process  surveillance  of  CSI  parts,  and 
imposed  Government  Source  Inspection  (GSI)  on  all  parts  that  have  been  identified  as 
CSIs  per  the  Joint  Service  CSI  Instruction 


2.  Conduct  periodic  CSI  Program  evaluations  of  Lockheed  Martin  and  their  suppliers  lo  ensure 
compliance  with  public  law  and  the  Joint  Service  CSI  Instruction. 

F-35  JPO  Response:  Agree 

F-35  JPO  &  LM  conducted  a  CSI  TIM  in  Nov  2014  following  completion  of  the  DoD  IG 
Quality  Assurance  &  Corrective  Action  Evaluation  to  address  DoD  IG  “Opportunity  For 
Improvement"  comments  issued  for  CSIs  and  CSI  Program  Plan. 

F-35  JPO,  DCMA  &  LM  are  conducting  bi-weekly  telecoms  and  periodic  TIMs  on-site  at 
LM  until  SDD  contract  agreement  is  reached  to  implement  the  full  scope  CSI  Program 
i  n  f ra  stru  ctu  re  a  n  d  p  rocesse  s 

After  F-35  full  scope  CSI  implementation  is  underway,  the  F-35  JPO  will  transition  to  a 
monitoring  phase  with  periodic  reviews,  which  will  leverage  or  combine  with  other 
JPO/DCMA  activities  (joint  audits,  Production  Readiness  Reviews,  quality  reviews, 
metrics,  etcetera). 
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F-35  Joint  Program  Office  and  Defense 
Contract  Management  Agency  Response  to 
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Finding  B 

Oversight  of  Product  Development  and  Realization  of 
Requirements  Was  Inadequate 

JPO  was  not  adequately  managing  the  F-35  Program  Lo  ensure  LhaLal!  system-level  requirements 
and  capabilities  delmed  in  the  fSF  JCS  were  realized  and  verified,  Asa  result,  delivered  and  future 
aircraft  may  not  satisfy  all  system  capability  requirements  and  it  is  unclear  to  what  extent  these 
capabilities  will  be  achieved  by  the  completion  of  the  SDD  phase, 

Discussion 

The  JCS  document  identifies  more  than  400  system-level  requirements  that  define  performance 
characteristics.  JPO  and  Lockheed  Martin  jointly  identified  21  system  level  requirements  contained 
wiLhin  the  JCS  that  were  unlikely  to  be  achieved  by  the  end  of  the  SDD  phase  of  Lhe  program. 
However,  JPO  and  Lockheed  Martin  did  not  consider  these  21  requirements  as  risks  and  did  not 
handle  them  in  accordance  with  their  risk  management  process. 

In  addition,  JPO  still  has  not  defined  objective  and  measureable  aircraft  acceptance  criteria  for  each 
LRIP  lot  The  confirmation  criteria  defined  In  the  CCDD  did  not  include  clearly  measurable  and 
verifiable  acceptance  criteria  that  were  dearly  traceable  back  U>  the  JCS,  This  was  documented  in 
our  previous  evaluation  and  continues  to  be  a  challenge. 

F-35  JPO  Response:  Disagree 

The  F-35  JPO  has  progressively  made  improvements  to  aircraft  acceptance  criteria.  In 
LRIP  6,  as  in  previously  LRIPsr  known  JCS  non-compliances  were  documented  It  is 
true  that  at  aircraft  acceptance  F-35  JPO  does  not  require  IM  to  positively  verify 
compliance  to  every  JCS  requirement.  This  is  due  to  the  concurrent  nature  of  SDD  and 
Production  schedules  The  SDD  schedules  identify  when  each  JCS  requirement  is 
expected  to  meet  compliance  criteria.  The  vast  majority  of  compliance  occurs  in  the 
2016  and  2017  timeframe.  If  F-35  JPO  were  to  impose  JCS  compliance  assessments 
in  production,  prior  to  expected  SDD  JCS  compliance  dates,  the  assessments  would  be 
based  on  subjective  rather  than  objective  criteria  because  proper  SDD  certification 
evidence  would  not  be  available.  The  F-35  JPO  deemed  that  subjective,  premature 
JCS  compliance  assessments  were  not  value  add  for  the  program.  Additionally,  these 
subjective  assessments  would  negatively  impact  SDD  resources  (people,  labs  and  flight 
test  assets)  and  schedules 


Recommendations 

B,  We  recommend  that  the  F-35  Joint  Program  Office; 

1.  Clearly  define  contractual  criteria  for  Lhc  acceptance  of  all  future  and  fielded  aircraft  to  ensure 
that  aircraft  capabilities  are  verified. 
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F-35  Joint  Program  Office  and  Defense 
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F-35  Lightning  II  Program  Quality  Assurance  and  Corrective  Action  Evaluation 
Project  No.  D2  01 4- DTOT AD-0004.000 
F-35  JPO  and  DCMA  Findings,  Recommendations  and  Responses 

F-35  JPO  Response:  Agree 

F-35  JPO  is  progressively  refining  acceptance  criteria  with  each  LRIP  lot.  The  use  of 
CCDDs  will  continue  to  be  used  for  LRIP  7  and  LRIP  8.  These  CCDD  items  require 
positive  verification  activity  to  occur,  often  with  dedicated  flight  test  evaluations;  and 
they  provide  a  means  to  ensure  the  weapons  system  meets  warfighter  expectation  for 
that  production  lot.  However,  the  use  of  these  documents  does  not  negate  the 
responsibility  to  document  any  known  JCS  non-compliances  at  the  time  of  aircraft 
acceptance.  This  will  continue  to  occur  as  it  has  occurred  in  the  past.  As  SDD 
progresses  F-35  JPO  does  expect  more  qualification  evidence  to  be  available  which  will 
allow  for  more  robust  documentation  of  JCS  non-compliance  at  the  time  of  aircraft 
acceptance.  This  will  be  done  in  addition  to  requiring  positive  verification  for  all  CCDD 
items. 

Beginning  with  LRIP  9,  the  next  lot  to  be  negotiated,  F-35  JPO  will  require  full  JCS 
compliance.  F-35  JPO  will  assess  LRIP  9+  product  acceptance  against  the  SDD 
specification.  For  future  full-rate  production  contracts,  there  is  planned  a  separate 
production  specification  that  will  leverage  the  final  SDD  specification 

2.  Ensure  that  all  21  system-level  requirements,  in  addition  to  the  risks  associated  with  these 
capabilities,  are  documented,  tracked,  and  mitigated  using  the  established  risk  management 
process. 

F-35  JPO  Response:  Partially  Agree 

F-35  JPO  sees  merit  in  tracking  to  closure  the  contractually  defined  21  system  level 
requirements  that  may  not  be  fully  met.  F-35  JPO  agrees  the  high-level  visibility  of  a 
formal  tracking  Action  Item  to  periodically  status  these  requirements  to  closure  will 
ensure  resolution  of  them  in  an  expeditious  manner. 

However,  F-35  JPO  DISAGREES  with  the  recommendation  to  track  these  21  system 
level  requirements  through  the  risk  management  process.  These  21  requirements  are 
not  risks  -  they  are  a  subset  of  requirements  where  the  program  has  jointly 
acknowledged  that  the  level  of  specified  performance  will  not  be  met.  Conversely,  the 
purpose  of  risk  management  is  to  highlight  and  take  proactive  actions  to  avert  a 
potentially  bad  outcome,  such  as  a  specification  miss.  Within  the  F-35  risk 
management  process,  risks  are  managed  via  a  series  of  funded  mitigation  steps  and 
technical  gates  to  eliminate  or  reduce  the  likelihood  and/or  consequence  of  each  risk. 
With  these  21  requirements,  F-35  JPO  has  already  acknowledged  the  specification 
misses  and  has  a  contractual  agreement  to  spend  no  more  money  to  improve  the 
design  -  using  the  cost  as  an  independent  variable  principle  for  the  requirements.  The 
only  remaining  program  resources  associated  with  these  21  requirements  is  for 
gathering  qualification  evidence  to  verify  the  capability  of  the  design  as  is.  When 
qualifications  are  complete,  F-35  JPO  will  reset  the  specification  requirements  to  the 
achieved  performance  and  negotiate  consideration  for  each  of  the  shortfalls. 
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A  closure  tracking  /  status  Action  Item  for  these  21  system  level  requirements  would 
regularly  assess  progress  toward  reconciliation  and  closure  of  the  specification  to 
realized  performance:  qualification  actions,  coordination  of  verification  evidence,  draft 
specification  change  request  engineering  review.  Configuration  Steering  Board 
approval,  Change  Control  Board  approval  and  contract  negotiations  /  contract 
modifications.  However,  for  these  21  system  level  requirements,  neither  risk  mitigations 
nor  design  changes  are  funded  within  the  remaining  scope  of  the  System  Design  and 
Development  contract  As  an  aside,  several  of  the  21  system  level  requirements  are 
separately  tracked  as  Program  Technical  Performance  Measures 


Finding  C 

The  Quality  Assurance  Organization  Was  Not 
Independent  and  Not  Adequately  Staffed 

jPO  had  not  defined  its  quality  assurance  organization's  roles  and  responsibilities  and  had  not 
provided  adequate  staffing  to  perform  effective  oversight  of  the  F-35  Program. 

Also,  the  quality  assurance  organization  reports  to  the  Director  of  Production  rather  than  directly 
Lo  the  F‘35  PEO.  The  |PO  quality  assurance  function  did  not  have  the  necessary  independence  to 
implement  the  quality  management  mission  far  development  production,  and  sustainment  ol The 
system.  A  quality  assurance  organization  that  lacks  independence  or  the  inhcrenL  authority  to 
enforce  quality  management  requirements  may  not  effectively  mitigate  cost,  schedule,  and 
performance  risks  to  the  F-35  Program. 

Discussion 

Our  previous  evaluation  recommended  that  J PC  establish  an  independent  quality  assurance 
organization.  ]P0  initiated  the  establishment  of  a  quality  organization,  assigned  a  Quality  Lead,  and 
plans  to  further  develop  and  staff  the  organization,  But,  the  organization  reports  to  the  Director  of 
Production  rather  than  directly  to  the  F-35  PEO,  Because  the  desire  to  meet  production  schedules 
can  often  prevent  adequate  or  sufficient  root  cause  analysis  and  corrective  action,  a  quality 
assurance  organization  that  is  independent  and  reports  to  the  PEO  ensures  greater  influence  in 
program  decisions.  The  current  organizational  structure  may  limit  the  quality  assurance 
organization's  ability  to  objectively  report  program  qualiLy  management  performance. 

F-35  JPO  Response:  Partially  Agree 

The  F-35  JPO  does  not  place  schedules  ahead  of  quality.  Further,  the  F-35  JPO  has 
withheld  delivery  of  aircraft  to  assure  quality  issues  are  addressed.  Although  the  F-35 
JPO  deems  the  current  reporting  structure  adequate  to  meet  program  needs,  it  has 
agreed  to  create  a  more  robust  quality  organization  to  ensure  quality  management  is 
objectively  measured  and  reported. 

Recommendation 
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C.  Wo  recommend  that  the  F-35  Joint  Program  Office  realign  Lhc  quality  assurance  organization  La 
report  directly  to  the  PEO,  define  the  organizations  roles  and  responsibilities,  and  staff  the 
organization  appropriately. 

F-35  JPO  Response:  Agree 

Efforts  to  establish  an  independent  quality  organization  are  on  track  As  documented  in 
F-35  JPCFs  August  2014  response  and  reiterated  during  presentations  and  discussions 
with  DODIG  team  members.  A  quality  organization  will  be  in  place  12-24  months  from 
the  May  2014  decision. 


Finding  D 

Reduction  of  the  Assembly  Defect  Rate  Was 
inadequate 

JPO  and  DCMA  management  were  not  committed  to  ensuring  that  assembly  defect  rate  objectives 
and  goals  were  realized.  Lack  of  ability  to  reduce  defects  and  the  scrap,  rework,  and  repair  rate  in 
accordance  with  defined  program  objectives  will  negatively  impact  program  cost  and  schedule, 

Discussion 

JPO  was  not  ensuring  that  Lockheed  Martin  was  fulfilling  its  commitment  to  a  90-percent  assembly 
defect  rate  reduction  per  aircraft  to  meet  the  full-rate  production  goal  of  completing  one  aircraft 
per  day.  We  were  not  able  to  determine  Lockheed  Martin's  progress  in  meeting  its  commitment  to  a 
90- percent  reduction  in  the  assembly  defect  rate  because  It  was  not  tracking  Its  progress  using  the 
correct  data  for  5DD  or  LRIP  lot  deliveries.  In  accordance  with  conflicting  information  contained  in 
the  quality  management  plan,  Lockheed  Martin  was  tracking  the  number  of  nonconforming 
material  records  rather  than  the  number  of  assembly  delects.  Lockheed  Martin  demonstrated  a  23- 
percent  reduction  in  nonconforming  material  records  compared  to  the  60- percent  defect  rate 
reduction  that  should  have  been  realized  after  LRIP  lot  5,  in  addition,  |PO  and  DCMA  did  not  ensure 
LbaL  corrective  actions  for  product  nonconformances  were  adequately  implemented  to  reduce  the 
assembly  defecL  rate.  At  the  time  of  our  evaluation,  we  were  not  aware  of  any  initiatives  or  value 
engineering  change  proposals  specifically  aimed  at  the  reduction  of  the  assembly  defect  rate  in 
accordance  with  the  program  quality  management  plan  goals. 

F-35  JPO  Response:  Disagree 

The  program  quality  90%  objective  the  DODIG  references  was  a  ratio  of  Scrap  Rework 
and  Repair  (SRR)  to  production  labor  hours  (hence  a  Tate1},  This  is  a  normalization 
technique  to  assess  levels  of  SRR  relative  to  volume  of  work  in  a  given  area.  This 
normalization  is  a  valuable  metric;  however,  it  is  not  the  only  metric.  In  the  current  case 
of  F-35  the  production  line  which  is  still  experiencing  significant  production  labor  hour 
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reductions  as  a  result  of  progressing  down  predictable  teaming,  the  ratio  of  declining 
SRR  hours  to  declining  production  labor  hours  has  remained  relatively  steady.  The 
F-35  JPO,  DCMA  and  LM  have  worked  to  understand  this  phenomenon  and  analyzed 
other  metrics  to  evaluate  the  program  s  progress  toward  reductions  necessary  to  reach 
high  production  rates  by  increasing  efficiency  and  flow, 

•  Factory  SRR  hours  have  decreased  65%  since  the  first  production  lot  Supplier 
responsible  SRR  has  been  reduced  80%  and  the  program  is  setting  a  goal  to  reduce 
the  current  level  another  25%  again  this  year 
»  Quality  Escapes  to  the  field  are  another  measure,  which  have  decreased  75%  since 
2011 


Recommendations 

D,  We  recommend  that  the  F-35  Joint  Program  Office; 

1  insure  LhaL  Lockheed  Marlin  implements  initiatives  La  reduce  the  assembly  deled  rate  of  the 
product  to  meet  planned  full  rate  production  goals. 

F-35  JPO  Response:  Agree 

As  presented,  Lockheed  Martin  has  initiated  several  initiatives  to  reduce 
nonconformances  of  all  types  both  within  their  facilities  and  throughout  their  supply 
chains.  The  F-35  JPO  is  committed  to  ensuring  quality  objectives  and  goals  are 
realized,  therefore  monitors  progress  closely  with  monthly  and  quarterly  quality  metrics 
reviews  and  Production  Readiness  Reviews  Additionally  F-35  JPO  production 
operations  and  quality  subject  matter  experts  attend  Systems  Engineering  Technical 
Reviews  to  assure  designs  are  producible  and  encompass  quality  objectives.  Additional 
improvement  incentives  are  being  realized  through  the  program  s  Blueprint  for 
Affordability  which  provides  a  cost  share  between  the  contractor  and  government  to 
fund  the  improvements,  while  not  solely  dedicated  to  quality  and  producibility 
improvements  the  majority  of  projects  achieve  savings  through  defect  reduction  by 
focusing  on  producibility.  F-35  JSPG  is  satisfied  with  the  methodology  LM  is  using  to 
monitor  SRR  in  lieu  of  a  ratio  to  production  hours.  Additional  metrics  are  being 
evaluated  by  F-35  JPOr  DCMA  and  the  contractors  independently  and  jointly  to 
continue  driving  discovery  of  nonconformances  at  their  source  (eliminate  escapes)  as 
well  as  facilitating  correction.  The  mutually  agreed  to  quality  goals  are  part  of  the  Joint 
Program  Commitments,  which  the  F-35  JPO  may  consider  implementing  in  contracts, 
though  they  are  not  currently. 

3.  Coordinate  with  DCMA  to  implement  an  effective  root  cause  analysis  and  corrective  action 
process  in  order  to  meet  assembly  defect  rate  goals. 
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F-35  JPO  Response:  Partially  Agree 

There  are  effective  Corrective  Action  processes  already  in  place  within  Lockheed  Martin 
and  the  supply  chain.  DCMA  and  F-35  JPO  quality  organizations  focus  heavily  on 
continually  improving  these  systems  and  process  to  assure  cause  analysis  leads  to  true 
root  cause  and  does  not  stop  with  direct  cause.  This  includes  areas  outside  production, 
including  work  instructions,  tooling ,  design,  facilities,  human  resources  and  even  within 
the  government  should  government  furnished  equipment  property  or  direction  be  a 
contributor  Lockheed  Martin  has  also  implemented  internal  corrective  action  processes 
for  non-shop  floor  continual  improvement  opportunities  While  F-35  JPO  agrees 
effective  root  cause  and  corrective  action  are  essential  to  improvement,  beyond 
continuous  high  priority  focus  through  Corrective  Action  Board  meetings,  metrics 
reviews,  DCMA  issuance  of  requests  for  corrective  action  no  additional  changes  are 
necessary  at  this  time. 


Finding  E 

Corrective  Action  Request  Escalation  Was  Inadequate 

DCMA  Fort  Worth  was  not  escalating  unresolved  CARS  to  the  next  higher  level  as  required  by  the 
applicable  DCMA  instruction.  If  DCMA  does  not  escalate  CARs  as  required,  deficiencies  may 
continue  to  occur  and  DCMA  and  ]PO  leadership  may  not  be  aware  of  problems  until  they  adversely 
impact  the  cost,  schedule,  and  performance  of  the  program. 

Discussion 

DCMA  was  not  following  Us  policy  to  escalate  CARs  to  the  next  higher  level  when  repetitive  Level  ll 
CARs  have  been  issued,  when  CAPs  were  rejected  and  not  resubmitted  within  the  IfLday 
requirement,  nr  when  the  corrective  action  validation  is  rejected.  We  identified  instances  when 
multiple  CARs  were  documented  for  the  same  violation,  but  none  were  escalated  to  the  next  level. 
We  also  noted  instances  when  CAPs  were  rejected  and  were  not  resubmitted  within  the  10-day 
requirement,  but  did  not  result  in  DCMA  escalating  the  CAR  to  the  next  level. 

DCMA  Response:  Agree 

DCMA  agrees  we  did  not  execute  to  DCMA  instruction  #1201  as  interpreted  by  the  DoD 
!G  team.  We  agree  that  DCMA  instruction  1201  requires  clarification  It  is  currently 
under  revision  to  address  clarification  to  the  CAR  efevation  process. 

DCMA  believes  this  risk  is  mitigated  based  on  a  comprehensive  program  quality 
strategy  between  DCMA,  F-35  JPO,  and  LM.  There  is  a  monthly  metrics  cadence  in 
which  the  DCMA  Quality  organization  provides  independent  CAR  analysis  The  data 
presented  in  these  meetings  provides  insight  of  non-compliances  identified  through 
DCMA  and  LM  Quality  surveillance  activities. 


Recommendations 
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E.  Wo  recommend  that  the  Defense  Contract  Management  Agency: 

1.  Review  all  unresolved  CARs  and  escalate  those  that  meet  the  criteria  established  in  DCMA 
policies  and  instructions. 

DCMA  Response:  Agree 

DCMA  agrees  we  did  not  execute  to  DCMA  Instruction  #1201  as  interpreted  by  the  DoD 
IG  team.  We  agree  that  DCMA  instruction  1201  requires  clarification.  It  is  currently 
under  revision  to  address  clarification  to  the  CAR  elevation  process.  DCMA 
Headquarters  will  provide  an  updated  policy  with  an  estimated  completion  date  of  early 
2015.  Upon  policy  update  approval,  DCMA  LMFW  will  review  all  open  CARS  to 
determine  if  any  warrant  elevation  to  the  next  level  as  recommended 

2,  Assess  all  CARs  that  were  not  properly  elevated  and  assess  any  impact  on  the  product 

DCMA  Response:  Agree 

DCMA  LMFW  will  assess  all  open  CARs  based  on  new  policy  guidance  to  identify, 
analyze,  and  assess  any  impacts  to  F-35  product 


Finding  F 

Software  Quality  Management  Was  Insufficient 

fPO  and  Lockheed  Martin  were  not  ensuring  software  quality  management  processes  such  as 
configuration  management,  failure  reporting  and  corrective  action,  nonconformance  root  cause  and 
analysis,  and  training  were  performed  sufficiently  to  prevent  delayed  mission  capability,  software 
rework,  and  possibly  additional  program  costs.  In  addition,  undiscovered  software  defects  could 
result  In  performance  degradation  or  even  catastrophic  failure  of  aircraft,  subsequently  resulting  in 
loss  of  aircraft  and  death. 

Discussion 

Lockheed  Martin's  software  development  methodology  did  not  facilitate  the  performance  of 
physical  and  functional  configuration  audits.  The  majority  of  new  aircraft  capability  is  based  on 
additional  software  builds:  therefore,  it  is  necessary  to  perform  functional  and  physical 
configuration  audits.  These  types  of  audits  ensure  that  CSCls  exist  and  are  configured,  tested,  and 
documented  in  accordance  with  software  specifications  and  requirements  to  mitigate  aircraft 
performance  and  reliability  risks.  In  addition,  deficiencies  in  Lockheed  Martin's  software 
development  training  program  may  lead  to  knowledge  gaps  which  could  result  in  software 
development  process  failures  and  software  defects.  Additionally,  the  classification  of  the  severity  of 
software  deficiencies  identifies  the  level  ol  verification  and  corrective  action  required.  Lockheed 
Martin's  failure  to  ensure  software  subcontractors,  such  as  Northrop  Grumman,  were  categorizing 
software  deficiencies  in  accordance  with  the  F-35  Air  System  Software  Development  Plan  does  not 
provide  confidence  in  delivered  software.  Allowing  open  deficiencies,  especially  those  whose 
severities  are  not  characterized  and  understood,  can  result  in  performance  degradation  or 
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catastrophic  failure  during  operation.  JPO  needs  to  ensure  that  rigorous  software  processes  exist  at 
Lockheed  Martin, 

F-35  JSPO  Response:  Disagree 

The  F-35  JPO  considers  F-35  software  quality  management  is  sufficient,  specifically  the 
air  vehicle  software  quality  referenced  in  the  draft  DOD  IG  audit  report.  The  F-35  JPO  is 
confident  current  air  vehicle  software  configuration  management,  failure  reporting  and 
corrective  action,  root  cause  analysis,  and  training  are  sufficient  and  adequate  to 
support  program  mission  capability  and  cost  objectives.  The  F-35  JPO  is  confident  in 
the  current  air  vehicle  software,  systems  engineering,  and  airworthiness  certification 
processes  will  ensure  no  undiscovered  software  defects  will  be  delivered  with  the  air 
vehicle  that  couid  result  in  performance  degradation  or  even  catastrophic  failure  of 
aircraft. 

Specific  DOD  IG  Finding  Responses: 

1  DOD  IG  Finding:  Lockheed  Martin's  software  development  methodology  did  not 
facilitate  the  performance  of  physical  and  functional  configuration  audits.  The 
majority  of  new  aircraft  capability  is  based  on  additional  software  builds; 
therefore,  it  is  necessary  to  perform  functional  and  physical  configuration  audits. 
These  types  of  audits  ensure  that  CSCIs  exist  and  are  configured,  tested,  and 
documented  in  accordance  with  software  specifications  and  requirements  to 
mitigate  aircraft  performance  and  reliability  risks. 

F-35  JPO  Response:  In  accordance  with  the  F-35  program  Software 
Development  Plan  and  Configuration  Management  Plan,  physical  and  functional 
configuration  audits  of  the  type  referenced  by  the  DOD  IG  report  are  conducted 
against  the  air  vehicle  Software  Data  Load,  which  is  a  compilation  of  vehicle  and 
mission  systems  components.  The  program  is  conducting  FCA/PCAs  in 
accordance  with  the  F-35  CM  Plan,  2YZA00002,  on  all  CIs  and  CSCIs  identified 
in  the  F-35  Cl  List,  2YZA0094T  approved  by  the  F-35  JPO,  This  finding 
erroneously  presumes  that  certain  software  items  which  are  not  identified  in  the 
Cl  list  are  CSCIs  and  therefore  must  be  subject  to  FCA/PCA  requirements 
Mission  System  and  Vehicle  System  computer  programs  are  elements  which 
contribute  to  the  composition  of  the  aircraft  Software  Data  Load.  It  is  the  aircraft 
Software  Data  Load  which  is  identified  in  the  F-35  Cl  List  as  a  CSCI,  and  it  is  this 
software  product  on  which  FCA/PCA  activity  will  be  conducted.  This  practice  is 
consistent  with  software  development  practices  on  all  weapon  systems 
developed  at  the  Fort  Worth  facility,  including  the  F-16  and  F-22,  in  accordance 
with  the  DoD  required  Single  Process  Initiative  for  the  Fort  Worth  Facility.  The 
F-35  program  has  a  very  rigorous  and  structured  verification  and  validation 
process  for  software  enabled  systems  capabilities,  including  the  identification  and 
management  of  all  software  components  that  contribute  to  safety  critical 
functions.  In  addition,  F-35  air  vehicle  software  undergoes  extensive  independent 
reviews  by  external  airworthiness  agencies  from  the  United  States  Air  Force  and 
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Navy  in  the  context  of  MIL-HDBK-516,  as  well  as  the  Ministries  of  Defense  of  the 
United  Kingdom  and  Australia.  These  agencies  have  independently  confirmed 
the  adequacy  of  the  physical  configuration  and  functional  ability  of  the  air  vehicle 
software  for  safe  operation  in  the  delivered  operational  envelope  The  F-35 
program  does  not  intend  to  modify  existing  functional  and  physical  configuration 
audit  practices  for  F-35  air  vehicle  software  based  on  this  finding. 

2.  DoD  IG  Finding:  Deficiencies  in  Lockheed  Martin's  software  development  training 
program  may  lead  to  knowledge  gaps  which  could  result  in  software 
development  process  failures  and  software  defects. 

F-35  JPO  Response:  Lockheed  Martin  has  a  structured  and 
accountable  training  program  for  all  software  engineers  in  the  LM 
aeronautical  division,  documented  in  PM-4002  {Software  Engineering 
Training).  In  addition,  the  LM  F-35  program  has  a  structured  and 
accountable  training  program  for  F-35  specific  software  development 
environments  and  tools.  Training  history  for  both  LM  Aero  and  F-35 
specific  training  were  provided  to  the  IG  team,  as  well  as  learning 
plans  for  specific  classes  of  software  engineering  disciplines.  Air 
vehicle  software  engineers  on  the  F-35  program  comprise  a  body  of 
world  class  talent  -  they  are  industry  experts  in  the  development  of 
hard  real  time  embedded  flight  and  mission  critical  software,  and  have 
in  most  cases  decades  of  experience  in  vehicle  and  mission  systems 
software  development  on  other  4lh  and  5th  generation  fighter  programs. 

They  are  sought  after  by  industry  to  provide  instruction  to  other 
engineers  in  their  discipline  areas,  and  work  directly  with  the 
developers  of  the  software  tools  and  environments  they  are  using  to 
incorporate  additional  enhancements  and  features  to  increase 
productivity  and  efficiency.  The  F-35  JPO  does  not  intend  to  pursue 
any  additional  training  initiatives  for  the  LM  software  development  team 
based  on  this  finding. 

3.  DoD  IG  Finding:  The  classification  of  the  severity  of  software  deficiencies 
identifies  the  level  of  verification  and  corrective  action  required.  Lockheed 
Martin's  failure  to  ensure  software  subcontractors,  such  as  Northrop  Grumman, 
were  categorizing  software  deficiencies  in  accordance  with  the  F-35  Air  System 
Software  Development  Plan  does  not  provide  confidence  in  delivered  software. 
Allowing  open  deficiencies,  especially  those  whose  severities  are  not 
characterized  and  understood,  can  result  in  performance  degradation  or 
catastrophic  failure  during  operation. 

F-35  JPO  Response:  System  and  software  deficiency  severity  classification  and 
tracking  is  managed  and  maintained  at  the  air  vehicle  system  level,  in 
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accordance  with  F-35  program  systems  engineering,  software  engineering,  and 
system  verification  plans  The  DoD  IG  team  assumed  subcontractors  would 
independently  classify  and  track  their  anomalies  using  the  same  criteria  as  the  air 
vehicle  teams,  which  is  not  the  defined  process  or  practice  Subcontractor 
deficiencies  are  reported  against  the  specification  requirements  levied  to  the 
subcontractors,  but  cannot  be  categorized  by  the  subcontractor  team  in  terms  of 
air  vehicle  severity,  as  only  the  air  vehicle  systems  engineering  teems  in  the 
vehicle  systems  end  mission  systems  functional  areas  have  the  required  breadth 
and  scope  of  system  functionality  and  safety  criticality  to  adequately  classify  the 
severity  of  component  anomalies  in  the  context  of  air  vehicle  operations  The 
F-35  JPO  reviewed  the  specific  concerns  highlighted  by  the  DoD  IG  with 
Northrop  Grumman  failure  reporting,  and  found  the  reporting  practice  consistent 
with  the  documented  air  vehicle  software,  systems  engineering,  and  air  vehicle 
verification  plan.  There  are  no  open  deficiencies  delivered  with  the  air  vehicle 
that  are  not  characterized  by  hazard  severity  and  understood  within  the  context 
of  mission  and  safety  criticality,  and  which  have  not  been  adjudicated  with  the 
F-35  JPO,  airworthiness  authorities,  and  operational  users.  The  F-35  JPO  does 
not  plan  to  modify  the  existing  software,  systems  engineering,  and  system 
verification  process  or  practices  based  on  this  finding. 


Recommendation 

F.  We  recommend  that  the  F-35  joint  Program  Office  work  with  Lockheed  Martin  to  ensure 
software  quality  management  systems  are  improved;  metrics  should  be  reported  on  a  periodic 
basis  [for  example,  monthly)  to  evaluate  process  improvement. 

F-35  JPO  Response:  Agree 

The  F-35  JPO  has  been  and  will  continue  to  work  with  Lockheed  Martin  to  ensure  the 
adequacy  and  improvement  of  software  quality  management  processes  and  practices 
Metrics  reflecting  software  quality  and  quality  improvement  objectives  have  been 
developed  and  are  being  reported  and  assessed  at  both  the  vehicle  systems  and 
mission  systems  team  levels  on  a  monthly  basis,  as  well  as  at  the  mr  vehicle  level  and 
production  quality  on  a  monthly  basis 


Finding  G 

Subcontract  Management  Procedures  Were  Still 
Inadequate 

I  PD,  DCMA,  and  Lockheed  Martin  were  not  ensuring  that  contract  requirements  were  adequately 
flowed  down,  contract  deliverables  were  evaluated,  and  minor  nonconformances  were  approved  by 
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the  proper  authority,  hack  ofsulTLcient  subcontract  management  activities  may  result  in  the 
acceptance  of  products  that  do  not  meet  intended  performance  and  reliability  requirements. 

Discussion 

DCMA  and  Lockheed  Martin  were  not  ensuring  that  the  purchase  orders  identified  the  specific 
revisions  of  the  technical  requirement  documentation  necessary  for  the  production,  testing,  and 
inspection  of  each  Stem  being  purchased  and!  changes  to  technical  requirements  were  contractually 
documented.  They  also  did  not  ensure  that  contractually  required  supplier-generated  data  items 
were  delivered  by  subcontractors  when  required.  We  found  no  evidence  of  any  appropriate  actions 
taken  when  subcontractors  did  not  comply  with  these  requirements,  in  addition,  fPO  and  Lockheed 
Marlin  were  not  ensuring  that  disposition  and  approval  of  minor  nonconformances  were  approved 
only  by  DCMA,  who  has  the  delegated  authority  for  approval  of  minor  nonconformances. 

Recommendations 

G,  1.  We  recommend  that  the  L-35  Joint  Program  Office  ensure  that  all  minor  nonconformances  are 
evaluated  and  approved  only  by  DCMA, 

F-35  JPO  Response;  Partially  Agree 

F-35  JPO  will  work  with  DCMA  to  assure  compliance  with  regulation  and  direction.  As 
part  of  an  existing  effort  to  validate  requirements  flow  down  accuracy,  minor 
nonconformance  approval  will  be  included.  Scrap  and  rework-to-design-definition  (so 
called  rework  to  blue-print)  dispositions  ultimately  result  in  conforming  product  being 
presented  to  the  government  for  acceptance  and  alleviate  the  requirement  of  further 
DCMA  review.  DCMA  has  and  will  continue  to  intercede  in  repetitive  scrap  and  rework 
In  the  case  of  repair  or  use-as-is  disposition,  DCMA  reviews  these  in  accordance  with 
regulation  and  instruction  as  it  results  in  nonconforming  product  being  tendered  for 
acceptance.  The  F-35  JPO  will  work  with  DCMA  to  assure  nonconformance 
evaluations  are  properly  reviewed  in  accordance  with  requirements  and  action  plans  put 
in  place  by  March  31 , 2015. 

G.2.  We  recommend  that  the  Defense  Contract  Management  Agency  ensure  that  Lockheed  Martin 
flows  down  the  appropriate  technical  requirements  to  its  subcontractors  and  receives  and 
evaluates  contract  deliverables  within  the  required  time  frames, 

DCMA  Response:  Agree 

DCMA  LM  will  assess  current  functional  surveillance  strategies  to  determine 
system/process  surveillance  gaps  regarding  Subcontract  Management  Procedures 
DCMA  LM  surveillance  strategies  will  be  revised  as  necessary  to  ensure  LM  processes 
are  adequate  to  provide  appropriate  flow  down  of  requirements  and  execution  of  those 
requirements  are  achieved  by  their  subcontractors. 
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LOCKHEED  WAR 


r7¥^ 


Dr  Alexander  Eksrr 

Vice  President,  Quality  &  Mission  Success 
Lockheed  Martin  Aeronautics 


DATE: 

TO: 

ATTN: 

SUBJECT: 

REFERENCE: 


13  January  2015 

F-35  Joint  Strike  Fighter  Program  Office 


Update  and  Follow  Up  to  2014  Department  of  Defense  Inspector  General 
Quality  System  Review  at  Lockheed  Martin  Aeronautics 

Discussion  with  Department  of  Defense  Inspector  General;  Please 
Consider  for  Inclusion  with  Response  to  F-35  Lightning  II  Program  Quality 
Assurance  and  Corrective  Action  Report 


To  the  F-35  Joint  Strike  Fighter  Program  Office: 

As  noted  during  the  latest  IG  review  in  September,  much  progress  is  being  made  at  Lockheed 
Martin  Aeronautics  on  the  F-35  program.  The  company  continues  to  work  the  issues  while 
launching  initiatives  to  ensure  an  enduring  focus  on  process  excellence  and  zero  defects. 
Following  the  recent  review  in  2014,  the  company  quickly  took  action  to  address  the  concerns 
identified  by  the  audit  team.  Lockheed  Martin  Aeronautics  in  partnership  with  DCMA  and  the 
JPQ  documented  observations  made  by  the  audit  team  with  internal  Corrective  Action  Requests 
(iCARs).  A  Forma!  Corrective  Action  Plan  (CAP)  was  developed  for  each  ICAR  allowing  Subject 
Matter  Experts  (SMEs)*  in  collaboration  with  local  DCMA  and  JPQ  representatives,  to  drive  robust 
root-cause  analysis,  close  systemic  gaps,  and  jointly  validate  effectiveness  for  each  finding.  In 
addition  to  these  corrective  actions,  the  JPQ  continues  to  evolve  and  strengthen  management 
oversight  to  ensure  our  momentum  continues.  Added  oversight  methods  Include  multiple  quality 
system  reviews,  metrics  reviews,  joint  audits  and  other  regularly  scheduled  events  that  are 
planned  one -year  in  advance  and  tracked  monthly. 

The  JPO  and  Lockheed  Martin  Aeronautics  have  leveraged  the  IG  reviews  to  improve  the 
company  Quality  Management  System  with  several  changes  including  the  following: 

1 .  Review  formats,  content,  and  accountabilities  were  reshaped  while  ensuring  broader 
i  n clusi o  n  of  DC  M  A  a  n  d  othe  r  prog  ra  m  matic  sta  keh  olders ,  I  n  20 1 4  a  lone  over  20 
enhanced  quality  reviews  were  held  between  DCMA,  JPO,  Lockheed  Martin  Aeronautics, 
and  key  suppliers/  co-producers.  Many  of  these  events  will  continue  to  be  conducted 
weekly  and  monthly  to  drive  improvements  while  collecting  the  real-time  status  of  quality 
performance  and  addressing  program  quality  challenges. 

2 .  Th  e  G  lobal  Qu  a  lity  Co  u  noil  (G  QC )  was  created  to  s  u  pp  o  rt  supply  c  hai  n  i  m  prove  merits  a  nd 
share  best  practices  between  industry  partners.  The  GQC  is  chaired  by  the  Lockheed 
Martin  Aeronautics  Vice  President  of  Q&MS  and  represents  the  top  seventeen  mission 


70 


DODIG-2015-092 


Management  Comments 


Lockheed  Martin  Letter  to  F-35  Joint  Program  Office 
on  Quality  Improvements  (cont'd) 


critical  major  suppliers  and  co-producers  for  F-35,  The  GQC  is  actively  engaged  in 
improvement  and  quality  transformations  across  the  supply  chain.  During  2014,  the 
council  facilitated  joint  engagements  to  address  common  problem  sub-tier  suppliers,  offer 
training  exchanges  between  GQC  members,  improved  response  times  to  address 
challenges  between  companies,  and  execute  resource  exchanges  to  address  technical 
and  quality  issues, 

3.  The  Aeronautics  Quality  Transformation  (AQT)  was  established  to  further  Lockheed 
M arti n  Aero nautics’  focus  a n  quai ity  i m pro veme nt  Cham pio ned  by  the  bu s Eness 
president  and  sponsored  by  senior  executives,  the  focus  of  the  AQT  is  on  cultivating 
systematic  accountability  to  quality  across  the  company  and  value  chain.  AQT 
challenges  each  section  of  the  company  to  identify  areas  of  improvement  and  investments 
that  will  drive  out  defects.  Many  of  the  improvements  noted  during  the  DoD  IG  audit  and 
positive  changes  at  Lockheed  Martin  Aeronautics  are  directly  tied  to  the  corporate 
sponsored  AQT  efforts  and  projects, 

4.  Fort  Worth's  FOD  Prevention  Center  was  erected  as  a  state-of-the-art,  hands-on  learning 
clinic  for  FOD  detection  and  prevention.  The  center  has  provided  an  excellent  venue  for 
training  including  FOD  detection  and  retrieval,  enhanced  specialized  equipment  training, 
and  a  new  simulation  trainer.  The  annual  compliance  training  "Fundamentals  of  FOD 
Prevention1 2 3 4 5 6'  has  been  refreshed  to  deliver  improved  learning.  Process  compliance 
assessments  are  performed  by  the  FOD  prevention  team  across  all  factory  build  areas. 
Although  FOD  continues  to  be  a  challenge  for  Lockheed  Martin  Aeronautics  and  industry, 
real  improvements  have  been  established  and  are  yielding  results  across  the  enterprise, 

5.  Learning  and  development  of  human  capital  continue  to  be  key  investments  toward 
continuous  improvement  Investments  including  the  Mission  Success  Development 
Training  Initiative  (MSDTi),  training  and  certification  by  external  institutes,  and  the  quality 
engineering  transformation,  were  instituted  to  strengthen  employee  understanding  and 
ability  in  Quality  Management  and  Science  across  Lockheed  Martin  Aeronautics,  MSTDi 
is  a  twelve  month  training  curriculum  applying  the  principles  of  quality.  More  than  200 
students  from  various  functions  at  the  company  and  its  suppliers  have  completed  the 
MSTDi  program  and  40  percent  of  quality  engineers  are  certified  by  the  American  Society 
fo  r  Qu  a!  ity.  1 00  pe  roe  nt  ( 1 54)  of  Q  &  M  S  leade  rsh  i  p  an  d  sa  lari  e  d  em  p  loyee  s  have 
completed  AS9100  training,  and  100  percent  of  inspectors  have  completed  a  “Q101” 
training  course. 

The  focus  areas  mentioned  are  just  a  few  examples  of  the  work  underway.  These  efforts  are 
yielding  results  with  a  more  mature  Quality  Management  System,  product  quality  Improvements, 
and  program  momentum.  Examples  of  the  key  indicators  of  quality  improvement  include  the 
following; 

1 .  A  50  percent  reduction  in  defects  from  the  top  22  F-35  suppliers  during  2014; 

2.  80  percent  reduction  in  supplier  Scrap,  Rework,  and  Repair  (SRR)  since  LRtP  1 ; 

3.  Factory  SRR  hours  have  decreased  65  percent  between  LRIP  1  and  LRIP  6; 

4.  Production-caused  escapes  to  the  field  are  down  75  percent  since  2011; 

5.  Zero  FOD  defects  at  delivery  in  2014; 

6.  Flight  test  performance  continues  to  accelerate  and  validate  the  capability  of  the  platform. 


DODIG-2015-092 


71 


Management  Comments 


Lockheed  Martin  Letter  to  F-35  Joint  Program  Office 
on  Quality  Improvements  (cont'd) 


Difficult  early  program  technical  challenges  are  being  resolved  in  partnership  with  Lockheed 
Martin  Aeronautics,  the  JPO,  and  program  suppliers.  Contracting  challenges  are  being  diligently 
worked  and  resolved.  Affordability  continues  to  be  a  focus  area  with  multiple  efforts  resulting  in 
reduced  costs  as  evident  in  LRIP  over  LRIP  price  reductions. 

The  Lockheed  Martin  Aeronautics  continuous  improvement  strategy  is  being  realized  in  a  way 
that  is  benefiting  the  F-35  program  as  well  as  the  other  platforms  and  sites  that  make  up  the 
business.  The  assessments  by  the  DoD  IG  have  further  validated  those  improvements  and 
areas  of  needed  focus. 
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APUC  Average  Procurement  Unit  Cost 
AS  Aerospace  Standard 
CAP  Corrective  Action  Plan 
CAR  Corrective  Action  Request 
CCDD  Capability  Description  Document 
CSCI  Computer  Software  Configuration  Item 
CSI  Critical  Safety  Item 
CTOL  Conventional  Takeoff  and  Landing 
CV  Carrier-Suitable  Variant 
DCMA  Defense  Contract  Management  Agency 
DFARS  Defense  Federal  Acquisition  Regulation  Supplement 

DMSMS  Diminishing  Manufacturing  Sources  and  Material  Shortages 
FCA  Functional  Configuration  Audit 
FOD  Foreign  Object  Debris 

FRACAS  Failure  Reporting,  Analysis,  and  Corrective  Action  System 
HWCI  Hardware  Configuration  Item 
ICA  Integrated  Corrective  Action 

ICNI  Integrated  Communications,  Navigation,  and  Identification 
IG  Inspector  General 
JCS  JSF  Contract  Specification 
JPO  Joint  Program  Office 
JSF  Joint  Strike  Fighter 
LRIP  Low-Rate  Initial  Production 
OFI  Opportunity  for  Improvement 
PAUC  Program  Acquisition  Unit  Cost 
PCA  Physical  Configuration  Audit 
PEO  Program  Executive  Officer 
SDD  System  Development  and  Demonstration 
SDP  Software  Development  Plan 
SPAR  Software  Problem  Anomaly  Report 
STOVL  Short  Takeoff  and  Vertical  Landing 
UTAS  United  Technologies  Corporation,  Aerospace  Systems 
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U.S.  Department  of  Defense 


The  Whistleblower  Protection  Enhancement  Act  of  2012  requires 
the  Inspector  General  to  designate  a  Whistleblower  Protection 
Ombudsman  to  educate  agency  employees  about  prohibitions 
on  retaliation,  and  rights  and  remedies  against  retaliation  for 
protected  disclosures.  The  designated  ombudsman  is  the  DoD  Hotline 
Director.  For  more  information  on  your  rights  and  remedies  against 
retaliation,  visit  www.dodig.mil/programs/whistleblower. 


For  more  information  about  DoD  IG 
reports  or  activities,  please  contact  us: 

Congressional  Liaison 

congressional@dodig.mil;  703.604.8324 

Media  Contact 

public.affairs@dodig.mil;  703.604.8324 

Monthly  Update 

dodigconnect-request@listserve.com 

Reports  Mailing  List 

dodig_report@listserve.com 

Twitter 

twitter.com/DoD_IG 

DoD  Hotline 

dodig.mil/hotline 
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